OpenSearch indexes failing to move to cold storage

[jaskaransarkaria]

Background

We saw this issue on elastic search and we should get to the bottom of this before it becomes a problem but some indexes are failing to move out of warm storage. The metadata gets copied over but not all the data. This then fails as it thinks the index has already been moved.

Figure out why this might be happening or write a script that we run on a corn to clear out the failed metadata in cold

{
    "cause": "Failed to migrate index [live_kubernetes_cluster-2024.01.30] to cold storage because another index with the same name already exists in cold storage.",
    "message": "Failed to start cold migration [index=live_kubernetes_cluster-2024.01.30]"
}

@jaskaransarkaria has the beginnings of this script