From f2eec049aaed208ede1dfad2ac78df897d87582a Mon Sep 17 00:00:00 2001 From: Vishal Raj Date: Fri, 8 Nov 2024 13:13:20 +0000 Subject: [PATCH 1/6] Add reference to prefer-ilm setting for apm lifeycle known bug --- docs/en/observability/apm/known-issues.asciidoc | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/docs/en/observability/apm/known-issues.asciidoc b/docs/en/observability/apm/known-issues.asciidoc index 72f8990b39..df8a4eeb4c 100644 --- a/docs/en/observability/apm/known-issues.asciidoc +++ b/docs/en/observability/apm/known-issues.asciidoc @@ -91,7 +91,10 @@ used. One of the following approaches can be adopted to fix the unmanaged indice . Manually delete the indices when they are no longer needed. . Explicitly configure APM data streams with the default data stream lifecycle config. Using this approach would migrate all data streams to use data stream lifecycles, -which should be equivalent to the default ILM policies: +which should be equivalent to the default ILM policies. Note that if you have a custom +lifecycle policy in place for any of the datastreams then you don't need to do anything +for those datastreams as they are not affected by this bug. Going forward, for using +custom ILM policies with new datastreams, please consult the {ref}/apm-ilm-how-to.html[updated guide]. + [source,txt] ---- From b6e7c700e666eef63896dde804e4173971c00530 Mon Sep 17 00:00:00 2001 From: Vishal Raj Date: Fri, 8 Nov 2024 14:44:28 +0000 Subject: [PATCH 2/6] Refactor --- .../observability/apm/known-issues.asciidoc | 57 +++---------------- 1 file changed, 9 insertions(+), 48 deletions(-) diff --git a/docs/en/observability/apm/known-issues.asciidoc b/docs/en/observability/apm/known-issues.asciidoc index df8a4eeb4c..99ba057f2d 100644 --- a/docs/en/observability/apm/known-issues.asciidoc +++ b/docs/en/observability/apm/known-issues.asciidoc @@ -89,61 +89,22 @@ indices created in version 8.15.0 would remain unmanaged if the default ILM poli used. One of the following approaches can be adopted to fix the unmanaged indices: . Manually delete the indices when they are no longer needed. -. Explicitly configure APM data streams with the default data stream lifecycle config. -Using this approach would migrate all data streams to use data stream lifecycles, -which should be equivalent to the default ILM policies. Note that if you have a custom -lifecycle policy in place for any of the datastreams then you don't need to do anything -for those datastreams as they are not affected by this bug. Going forward, for using -custom ILM policies with new datastreams, please consult the {ref}/apm-ilm-how-to.html[updated guide]. +. Explicitly configure APM data streams with the default data stream lifecycle config +for the data streams configured to use default ILM policies. Using this approach would +migrate all affected data streams to use data stream lifecycles, which should be equivalent +to the default ILM policies. Note that the fix should be applied only to data streams that +are resulting in unmanaged indices due to default ILM policies. + [source,txt] ---- -PUT _data_stream/traces-apm-*/_lifecycle +PUT _data_stream/{{data_stream_name}}-{{data_stream_namespace}}/_lifecycle { - "data_retention": "10d" -} - -PUT _data_stream/traces-apm.rum*/_lifecycle -{ - "data_retention": "90d" -} - -PUT _data_stream/traces-apm.sampled*/_lifecycle -{ - "data_retention": "1h" -} - -PUT _data_stream/metrics-apm.*.1m-*/_lifecycle -{ - "data_retention": "90d" -} - -PUT _data_stream/metrics-apm.*.10m-*/_lifecycle -{ - "data_retention": "180d" -} - -PUT _data_stream/metrics-apm.*.60m-*/_lifecycle -{ - "data_retention": "390d" -} - -PUT _data_stream/metrics-apm.internal-*/_lifecycle -{ - "data_retention": "90d" -} - -PUT _data_stream/metrics-apm.app.*/_lifecycle -{ - "data_retention": "90d" -} - -PUT _data_stream/logs-apm.*/_lifecycle -{ - "data_retention": "10d" + "data_retention": } ---- +Default `` for each data stream is available in {observability-guide}/apm-ilm-how-to.html[this guide]. + // Link to fix if it exists This issue is fixed in 8.15.1 (https://github.com/elastic/elasticsearch/pull/112432[elastic/elasticsearch#112432]). From 3de4736707f02577b794cb696d2dfe18a4f06bcf Mon Sep 17 00:00:00 2001 From: Francesco Gualazzi Date: Fri, 8 Nov 2024 16:31:49 +0100 Subject: [PATCH 3/6] APM known issue correction --- docs/en/observability/apm/known-issues.asciidoc | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/en/observability/apm/known-issues.asciidoc b/docs/en/observability/apm/known-issues.asciidoc index 99ba057f2d..2535409b05 100644 --- a/docs/en/observability/apm/known-issues.asciidoc +++ b/docs/en/observability/apm/known-issues.asciidoc @@ -25,9 +25,10 @@ _Versions: XX.XX.XX, YY.YY.YY, ZZ.ZZ.ZZ_ == Upgrading to v8.15.x may cause ingestion to fail _Elastic Stack versions: 8.15.0+_ +_Fixed in Elastic Stack version 8.15.4_ // The conditions in which this issue occurs -The issue only occurs when _upgrading_ the {stack} from 8.12.2 or lower directly to any 8.15.x version. +The issue only occurs when _upgrading_ the {stack} from 8.12.2 or lower directly to any 8.15.x version prior to 8.15.4. The issue does _not_ occur when creating a _new_ cluster using any 8.15.x version, or when upgrading from 8.12.2 to 8.13.x or 8.14.x and then to 8.15.x. From d226ee9efb2d6a740ff535960d2ec1c4c70beb36 Mon Sep 17 00:00:00 2001 From: Brandon Morelli Date: Fri, 8 Nov 2024 09:21:44 -0800 Subject: [PATCH 4/6] Update docs/en/observability/apm/known-issues.asciidoc Co-authored-by: Colleen McGinnis --- docs/en/observability/apm/known-issues.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/en/observability/apm/known-issues.asciidoc b/docs/en/observability/apm/known-issues.asciidoc index 2535409b05..bf5cbfe75d 100644 --- a/docs/en/observability/apm/known-issues.asciidoc +++ b/docs/en/observability/apm/known-issues.asciidoc @@ -24,7 +24,7 @@ _Versions: XX.XX.XX, YY.YY.YY, ZZ.ZZ.ZZ_ [discrete] == Upgrading to v8.15.x may cause ingestion to fail -_Elastic Stack versions: 8.15.0+_ +_Elastic Stack versions: 8.15.0, 8.15.1, 8.15.2, 8.15.3_ + _Fixed in Elastic Stack version 8.15.4_ // The conditions in which this issue occurs From ef8f699ecf9113138ce66bb8f5055d075da83c56 Mon Sep 17 00:00:00 2001 From: Brandon Morelli Date: Fri, 8 Nov 2024 09:33:56 -0800 Subject: [PATCH 5/6] clean up text --- .../observability/apm/known-issues.asciidoc | 19 +++++++++---------- 1 file changed, 9 insertions(+), 10 deletions(-) diff --git a/docs/en/observability/apm/known-issues.asciidoc b/docs/en/observability/apm/known-issues.asciidoc index bf5cbfe75d..29ffaad4ba 100644 --- a/docs/en/observability/apm/known-issues.asciidoc +++ b/docs/en/observability/apm/known-issues.asciidoc @@ -85,16 +85,15 @@ stream lifecycle configuration, such existing data streams become unmanaged for lifecycle configurations. // How to fix it -Upgrading to 8.15.1 should fix any new indices created for the data stream. However, -indices created in version 8.15.0 would remain unmanaged if the default ILM policy is -used. One of the following approaches can be adopted to fix the unmanaged indices: - -. Manually delete the indices when they are no longer needed. -. Explicitly configure APM data streams with the default data stream lifecycle config -for the data streams configured to use default ILM policies. Using this approach would -migrate all affected data streams to use data stream lifecycles, which should be equivalent -to the default ILM policies. Note that the fix should be applied only to data streams that -are resulting in unmanaged indices due to default ILM policies. +Upgrading to 8.15.1 resolves the lifecycle issue for any new indices created for APM data streams. +However, indices created in version 8.15.0 will remain unmanaged if the default ILM policy is in place. +To fix these unmanaged indices, consider one of the following approaches: + +. Manually delete the unmanaged indices when they are no longer needed. +. Explicitly configure APM data streams to use the default data stream lifecycle configuration. +This approach migrates all affected data streams to use data stream lifecycles, +maintaining behavior equivalent to the default ILM policies. +Apply this fix only to data streams that are currently unmanaged due to default ILM policies. + [source,txt] ---- From 37b338c503af7f8f95054ca810e2965c06d3be05 Mon Sep 17 00:00:00 2001 From: Brandon Morelli Date: Fri, 8 Nov 2024 11:11:28 -0800 Subject: [PATCH 6/6] Update docs/en/observability/apm/known-issues.asciidoc Co-authored-by: Vishal Raj --- docs/en/observability/apm/known-issues.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/en/observability/apm/known-issues.asciidoc b/docs/en/observability/apm/known-issues.asciidoc index 29ffaad4ba..cf4308644c 100644 --- a/docs/en/observability/apm/known-issues.asciidoc +++ b/docs/en/observability/apm/known-issues.asciidoc @@ -93,7 +93,7 @@ To fix these unmanaged indices, consider one of the following approaches: . Explicitly configure APM data streams to use the default data stream lifecycle configuration. This approach migrates all affected data streams to use data stream lifecycles, maintaining behavior equivalent to the default ILM policies. -Apply this fix only to data streams that are currently unmanaged due to default ILM policies. +Apply this fix only to data streams that have unmanaged indices due to missing default ILM policies. + [source,txt] ----