From 54ea820af38dbfc534b491c251a4abcd5d0ca894 Mon Sep 17 00:00:00 2001 From: Mike Bell Date: Thu, 29 Aug 2024 12:06:56 +0100 Subject: [PATCH 1/3] docs: initial commit of elasticsearch runbook --- .../elasticsearch-storage-issues.html.md.erb | 29 +++++++++++++++++++ 1 file changed, 29 insertions(+) create mode 100644 runbooks/source/elasticsearch-storage-issues.html.md.erb diff --git a/runbooks/source/elasticsearch-storage-issues.html.md.erb b/runbooks/source/elasticsearch-storage-issues.html.md.erb new file mode 100644 index 00000000..907a7554 --- /dev/null +++ b/runbooks/source/elasticsearch-storage-issues.html.md.erb @@ -0,0 +1,29 @@ +--- +title: Elasticsearch Storage Issues +weight: 9999 +last_reviewed_on: 2024-08-29 +review_in: 6 months +--- + +# <%= current_page.data.title %> + +## Hot/Warm Node Parity + +A way of increasing the storage space for either Hot or Warm nodes is to increase them. If increasing one then the other must be equal as well, for example if upping the Hot nodes to 20 Warm should be upped to 20 as well. + +## Manually migrating failed indexes + +It may be necessary to migrate indexes manually. The process for this is: + +1. Navigate to "Policy managed indices" +2. Find `Failed to start **** migration` +3. Copy the index name +4. Search "Warm" indexes and get index size +5. Search "Cold" indexes and get index size +6. If the "Cold" index size is smaller than "Warm" then it can be deleted +7. Delete the index via the devtools `DELETE _cold/live_kubernetes_ingress-2024.07.13` +8. Click retry policy + +## Further Reading + +* [Open search best practices](https://runbooks.cloud-platform.service.justice.gov.uk/resolve-opensearch-shard-issues.html) \ No newline at end of file From a35c20cd60e4cfc8c80dd57bd79a3714fefc02f4 Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" Date: Thu, 29 Aug 2024 11:08:14 +0000 Subject: [PATCH 2/3] Commit changes made by code formatters --- runbooks/source/elasticsearch-storage-issues.html.md.erb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/runbooks/source/elasticsearch-storage-issues.html.md.erb b/runbooks/source/elasticsearch-storage-issues.html.md.erb index 907a7554..d9b0cffc 100644 --- a/runbooks/source/elasticsearch-storage-issues.html.md.erb +++ b/runbooks/source/elasticsearch-storage-issues.html.md.erb @@ -26,4 +26,4 @@ It may be necessary to migrate indexes manually. The process for this is: ## Further Reading -* [Open search best practices](https://runbooks.cloud-platform.service.justice.gov.uk/resolve-opensearch-shard-issues.html) \ No newline at end of file +* [Open search best practices](https://runbooks.cloud-platform.service.justice.gov.uk/resolve-opensearch-shard-issues.html) From 29b891bbc6741abfb9bac83a62d56cd2e109133c Mon Sep 17 00:00:00 2001 From: Mike Bell Date: Thu, 29 Aug 2024 12:35:06 +0100 Subject: [PATCH 3/3] docs: add section on shards --- runbooks/source/elasticsearch-storage-issues.html.md.erb | 2 ++ 1 file changed, 2 insertions(+) diff --git a/runbooks/source/elasticsearch-storage-issues.html.md.erb b/runbooks/source/elasticsearch-storage-issues.html.md.erb index 907a7554..fc0da6f4 100644 --- a/runbooks/source/elasticsearch-storage-issues.html.md.erb +++ b/runbooks/source/elasticsearch-storage-issues.html.md.erb @@ -11,6 +11,8 @@ review_in: 6 months A way of increasing the storage space for either Hot or Warm nodes is to increase them. If increasing one then the other must be equal as well, for example if upping the Hot nodes to 20 Warm should be upped to 20 as well. +The number of shards and the nodes should be increased by a multiple of the index with the largest number of shards, further [information](https://aws.amazon.com/blogs/opensource/open-distro-elasticsearch-shard-allocation/). + ## Manually migrating failed indexes It may be necessary to migrate indexes manually. The process for this is: