From 762e54fe77cec8bae4814d8f547df3562dd3c9ad Mon Sep 17 00:00:00 2001 From: Kostiantyn Masliuk <1pkg@protonmail.com> Date: Sat, 27 Jul 2024 17:27:50 -0700 Subject: [PATCH 1/8] initial apm-es-backpressure doc draft --- .../apm/apm-performance-diagnostic.asciidoc | 27 +++++++++++++++++++ 1 file changed, 27 insertions(+) create mode 100644 docs/en/observability/apm/apm-performance-diagnostic.asciidoc diff --git a/docs/en/observability/apm/apm-performance-diagnostic.asciidoc b/docs/en/observability/apm/apm-performance-diagnostic.asciidoc new file mode 100644 index 0000000000..44f572aa01 --- /dev/null +++ b/docs/en/observability/apm/apm-performance-diagnostic.asciidoc @@ -0,0 +1,27 @@ +[[apm-performance-diagnostic]] +=== APM Server performance diagnostic + +[[apm-es-backpressure]] +[float] +==== Diagnosing backpressure from {es} + +When {es} is under excessive load or indexing pressure, APM Server could experience the downstream backpressure when indexing new documents into {es}. +Most commonly backpressure from {es} will manifest itself in a form of higher indexing latency and/or rejected requests, which in return could lead to denied incoming requests by APM Server. + +To quickly identify possible issues try looking for similar error logs lines in APM Server logs: + +[source,json] +---- +... +{"log.level":"error","@timestamp":"2024-07-27T23:46:28.529Z","log.origin":{"function":"github.com/elastic/go-docappender/v2.(*Appender).flush","file.name":"v2@v2.2.0/appender.go","file.line":370},"message":"bulk indexing request failed","service.name":"apm-server","error":{"message":"flush failed (429): [429 Too Many Requests]"},"ecs.version":"1.6.0"} +{"log.level":"error","@timestamp":"2024-07-27T23:55:38.612Z","log.origin":{"function":"github.com/elastic/go-docappender/v2.(*Appender).flush","file.name":"v2@v2.2.0/appender.go","file.line":370},"message":"bulk indexing request failed","service.name":"apm-server","error":{"message":"flush failed (503): [503 Service Unavailable] "},"ecs.version":"1.6.0"} +... +---- + +To gain better insight into APM Server health and performance please consider enabling Elastic Stack monitoring by following the guide <>. +When enabled APM Server will additionally report a set of vital metrics to help you identify any performance degradation. +Most notably metric fields `beats_stats.metrics.libbeat.output.events.*` will depict the number of documents that are pending in the buffer for indexing, already indexed successfully or failed during the indexing. +See https://www.elastic.co/guide/en/beats/metricbeat/current/exported-fields-beat.html[{metricbeat} documentation] for the full list of exported metric fields. + +If rejected requests are identified try to follow the guide from https://www.elastic.co/guide/en/elasticsearch/reference/current/rejected-requests.html. +Alternatively try adjusting `max_retries` and `timeout` setting from <> to reduce APM Server indexing pressure. \ No newline at end of file From 1bf0aae2e1cef2341855a248d186c4697945ed99 Mon Sep 17 00:00:00 2001 From: Kostiantyn Masliuk <1pkg@protonmail.com> Date: Tue, 30 Jul 2024 15:07:45 -0700 Subject: [PATCH 2/8] address review comments --- .../apm/apm-performance-diagnostic.asciidoc | 18 +++++++++++++----- .../apm/troubleshoot-apm.asciidoc | 4 +++- 2 files changed, 16 insertions(+), 6 deletions(-) diff --git a/docs/en/observability/apm/apm-performance-diagnostic.asciidoc b/docs/en/observability/apm/apm-performance-diagnostic.asciidoc index 44f572aa01..6d1cd99a24 100644 --- a/docs/en/observability/apm/apm-performance-diagnostic.asciidoc +++ b/docs/en/observability/apm/apm-performance-diagnostic.asciidoc @@ -7,6 +7,7 @@ When {es} is under excessive load or indexing pressure, APM Server could experience the downstream backpressure when indexing new documents into {es}. Most commonly backpressure from {es} will manifest itself in a form of higher indexing latency and/or rejected requests, which in return could lead to denied incoming requests by APM Server. +As a result APM agents connected to the affected APM Server will suffer from throttling and/or request timeout when shipping APM events. To quickly identify possible issues try looking for similar error logs lines in APM Server logs: @@ -14,14 +15,21 @@ To quickly identify possible issues try looking for similar error logs lines in ---- ... {"log.level":"error","@timestamp":"2024-07-27T23:46:28.529Z","log.origin":{"function":"github.com/elastic/go-docappender/v2.(*Appender).flush","file.name":"v2@v2.2.0/appender.go","file.line":370},"message":"bulk indexing request failed","service.name":"apm-server","error":{"message":"flush failed (429): [429 Too Many Requests]"},"ecs.version":"1.6.0"} -{"log.level":"error","@timestamp":"2024-07-27T23:55:38.612Z","log.origin":{"function":"github.com/elastic/go-docappender/v2.(*Appender).flush","file.name":"v2@v2.2.0/appender.go","file.line":370},"message":"bulk indexing request failed","service.name":"apm-server","error":{"message":"flush failed (503): [503 Service Unavailable] "},"ecs.version":"1.6.0"} +{"log.level":"error","@timestamp":"2024-07-27T23:55:38.612Z","log.origin":{"function":"github.com/elastic/go-docappender/v2.(*Appender).flush","file.name":"v2@v2.2.0/appender.go","file.line":370},"message":"bulk indexing request failed","service.name":"apm-server","error":{"message":"flush failed (503): [503 Service Unavailable]"},"ecs.version":"1.6.0"} ... ---- -To gain better insight into APM Server health and performance please consider enabling Elastic Stack monitoring by following the guide <>. +To gain better insight into APM Server health and performance please consider enabling the monitoring feature by following the guide from <>. When enabled APM Server will additionally report a set of vital metrics to help you identify any performance degradation. -Most notably metric fields `beats_stats.metrics.libbeat.output.events.*` will depict the number of documents that are pending in the buffer for indexing, already indexed successfully or failed during the indexing. +Pay careful attention to the next metric fields: +- `beats_stats.metrics.libbeat.output.events.active` that represents the number of buffered pending documents waiting for indexing; +- `beats_stats.metrics.libbeat.output.events.acked` that represents the number of indexing operations that have completed successfully; +- `beats_stats.metrics.libbeat.output.events.failed` that represents the number of indexing operations that failed, it includes all failures; +- `beats_stats.metrics.libbeat.output.events.toomany` that represents the number of indexing operations that failed due to {es} responding with 429 Too many Requests; +- `beat.stats.output.elasticsearch.bulk_requests.active` that represents the number of bulk indexers available for making bulk index requests; +- `beat.stats.output.elasticsearch.bulk_requests.completed` that represents the number of already completed bulk requests; +- `beats_stats.metrics.output.elasticsearch.indexers.active` that represents the number of active bulk indexers that are concurrently processing batches; See https://www.elastic.co/guide/en/beats/metricbeat/current/exported-fields-beat.html[{metricbeat} documentation] for the full list of exported metric fields. -If rejected requests are identified try to follow the guide from https://www.elastic.co/guide/en/elasticsearch/reference/current/rejected-requests.html. -Alternatively try adjusting `max_retries` and `timeout` setting from <> to reduce APM Server indexing pressure. \ No newline at end of file +One likely cause of excessive indexing pressure or rejected requests is undersized {es}, to mitigate try following the guide from https://www.elastic.co/guide/en/elasticsearch/reference/current/rejected-requests.html. +Alternatively try adjusting `max_retries` and `timeout` settings from <> to reduce APM Server indexing pressure, note however that reducing retries may lead to a higher rate of dropped APM events. \ No newline at end of file diff --git a/docs/en/observability/apm/troubleshoot-apm.asciidoc b/docs/en/observability/apm/troubleshoot-apm.asciidoc index ee36b88fbc..ce51906aa2 100644 --- a/docs/en/observability/apm/troubleshoot-apm.asciidoc +++ b/docs/en/observability/apm/troubleshoot-apm.asciidoc @@ -54,4 +54,6 @@ include::apm-response-codes.asciidoc[] include::processing-performance.asciidoc[] -include::{observability-docs-root}/docs/en/observability/apm/debugging.asciidoc[] \ No newline at end of file +include::{observability-docs-root}/docs/en/observability/apm/debugging.asciidoc[] + +include::apm-performance-diagnostic.asciidoc[] \ No newline at end of file From e8de32d2f45dc9ed65ab53af571fe79cfd7b0d88 Mon Sep 17 00:00:00 2001 From: Kostiantyn Masliuk <1pkg@protonmail.com> Date: Tue, 30 Jul 2024 20:10:42 -0700 Subject: [PATCH 3/8] fix internal doc references --- docs/en/observability/apm/apm-performance-diagnostic.asciidoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/en/observability/apm/apm-performance-diagnostic.asciidoc b/docs/en/observability/apm/apm-performance-diagnostic.asciidoc index 6d1cd99a24..a641eaad12 100644 --- a/docs/en/observability/apm/apm-performance-diagnostic.asciidoc +++ b/docs/en/observability/apm/apm-performance-diagnostic.asciidoc @@ -19,7 +19,7 @@ To quickly identify possible issues try looking for similar error logs lines in ... ---- -To gain better insight into APM Server health and performance please consider enabling the monitoring feature by following the guide from <>. +To gain better insight into APM Server health and performance please consider enabling the monitoring feature by following the guide from <>. When enabled APM Server will additionally report a set of vital metrics to help you identify any performance degradation. Pay careful attention to the next metric fields: - `beats_stats.metrics.libbeat.output.events.active` that represents the number of buffered pending documents waiting for indexing; @@ -32,4 +32,4 @@ Pay careful attention to the next metric fields: See https://www.elastic.co/guide/en/beats/metricbeat/current/exported-fields-beat.html[{metricbeat} documentation] for the full list of exported metric fields. One likely cause of excessive indexing pressure or rejected requests is undersized {es}, to mitigate try following the guide from https://www.elastic.co/guide/en/elasticsearch/reference/current/rejected-requests.html. -Alternatively try adjusting `max_retries` and `timeout` settings from <> to reduce APM Server indexing pressure, note however that reducing retries may lead to a higher rate of dropped APM events. \ No newline at end of file +Alternatively try adjusting `max_retries` and `timeout` settings from <> to reduce APM Server indexing pressure, note however that reducing retries may lead to a higher rate of dropped APM events. \ No newline at end of file From d76bb446abacf62574febc79bf5cd07f02fcf2ab Mon Sep 17 00:00:00 2001 From: Kostiantyn Masliuk <1pkg@protonmail.com> Date: Wed, 31 Jul 2024 14:46:22 -0700 Subject: [PATCH 4/8] address review comments --- .../apm/apm-performance-diagnostic.asciidoc | 29 ++++++++++++------- 1 file changed, 18 insertions(+), 11 deletions(-) diff --git a/docs/en/observability/apm/apm-performance-diagnostic.asciidoc b/docs/en/observability/apm/apm-performance-diagnostic.asciidoc index a641eaad12..af35c14315 100644 --- a/docs/en/observability/apm/apm-performance-diagnostic.asciidoc +++ b/docs/en/observability/apm/apm-performance-diagnostic.asciidoc @@ -6,7 +6,7 @@ ==== Diagnosing backpressure from {es} When {es} is under excessive load or indexing pressure, APM Server could experience the downstream backpressure when indexing new documents into {es}. -Most commonly backpressure from {es} will manifest itself in a form of higher indexing latency and/or rejected requests, which in return could lead to denied incoming requests by APM Server. +Most commonly, backpressure from {es} will manifest itself in the form of higher indexing latency and/or rejected requests, which in return could lead APM Server to deny incoming requests. As a result APM agents connected to the affected APM Server will suffer from throttling and/or request timeout when shipping APM events. To quickly identify possible issues try looking for similar error logs lines in APM Server logs: @@ -19,17 +19,24 @@ To quickly identify possible issues try looking for similar error logs lines in ... ---- -To gain better insight into APM Server health and performance please consider enabling the monitoring feature by following the guide from <>. +To gain better insight into APM Server health and performance, consider enabling the monitoring feature by following the steps in <>. When enabled APM Server will additionally report a set of vital metrics to help you identify any performance degradation. Pay careful attention to the next metric fields: -- `beats_stats.metrics.libbeat.output.events.active` that represents the number of buffered pending documents waiting for indexing; -- `beats_stats.metrics.libbeat.output.events.acked` that represents the number of indexing operations that have completed successfully; -- `beats_stats.metrics.libbeat.output.events.failed` that represents the number of indexing operations that failed, it includes all failures; -- `beats_stats.metrics.libbeat.output.events.toomany` that represents the number of indexing operations that failed due to {es} responding with 429 Too many Requests; -- `beat.stats.output.elasticsearch.bulk_requests.active` that represents the number of bulk indexers available for making bulk index requests; -- `beat.stats.output.elasticsearch.bulk_requests.completed` that represents the number of already completed bulk requests; -- `beats_stats.metrics.output.elasticsearch.indexers.active` that represents the number of active bulk indexers that are concurrently processing batches; + +* `beats_stats.metrics.libbeat.output.events.active` that represents the number of buffered pending documents waiting for indexing; +(_if this value is increasing rapidly indicates {es} backpressure_) +* `beats_stats.metrics.libbeat.output.events.acked` that represents the number of indexing operations that have completed successfully; +* `beats_stats.metrics.libbeat.output.events.failed` that represents the number of indexing operations that failed, it includes all failures; +(_if this value is increasing rapidly indicates {es} backpressure_) +* `beats_stats.metrics.libbeat.output.events.toomany` that represents the number of indexing operations that failed due to {es} responding with 429 Too many Requests; +(_if this value is increasing rapidly indicates {es} backpressure_) +* `beats_stats.output.elasticsearch.bulk_requests.available` that represents the number of bulk indexers available for making bulk index requests; +(_if this value is equal to 0 indicates {es} backpressure_) +* `beats_stats.output.elasticsearch.bulk_requests.completed` that represents the number of already completed bulk requests; +* `beats_stats.metrics.output.elasticsearch.indexers.active` that represents the number of active bulk indexers that are concurrently processing batches; + See https://www.elastic.co/guide/en/beats/metricbeat/current/exported-fields-beat.html[{metricbeat} documentation] for the full list of exported metric fields. -One likely cause of excessive indexing pressure or rejected requests is undersized {es}, to mitigate try following the guide from https://www.elastic.co/guide/en/elasticsearch/reference/current/rejected-requests.html. -Alternatively try adjusting `max_retries` and `timeout` settings from <> to reduce APM Server indexing pressure, note however that reducing retries may lead to a higher rate of dropped APM events. \ No newline at end of file +One likely cause of excessive indexing pressure or rejected requests is undersized {es}. To mitigate this, follow the guidance in {ref}/rejected-requests.html[Rejected requests]. +If scaling {es} resources up is not an option, you can try to workaround by adjusting `flush_bytes`, `flush_interval`, `max_retries` and `timeout` settings described in <> to reduce APM Server indexing pressure. +However, consider that increasing number of buffered documents and/or reducing retries may lead to a higher rate of dropped APM events. \ No newline at end of file From f46701ab7284e1fb01bae37bb16a85099f2e6385 Mon Sep 17 00:00:00 2001 From: Kostiantyn Masliuk <1pkg@protonmail.com> Date: Wed, 31 Jul 2024 15:13:50 -0700 Subject: [PATCH 5/8] es backpressure troubleshoot doc fmt fix --- .../apm/apm-performance-diagnostic.asciidoc | 9 +++++---- docs/en/observability/apm/troubleshoot-apm.asciidoc | 1 + 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/docs/en/observability/apm/apm-performance-diagnostic.asciidoc b/docs/en/observability/apm/apm-performance-diagnostic.asciidoc index af35c14315..caab1a04f7 100644 --- a/docs/en/observability/apm/apm-performance-diagnostic.asciidoc +++ b/docs/en/observability/apm/apm-performance-diagnostic.asciidoc @@ -21,17 +21,18 @@ To quickly identify possible issues try looking for similar error logs lines in To gain better insight into APM Server health and performance, consider enabling the monitoring feature by following the steps in <>. When enabled APM Server will additionally report a set of vital metrics to help you identify any performance degradation. + Pay careful attention to the next metric fields: * `beats_stats.metrics.libbeat.output.events.active` that represents the number of buffered pending documents waiting for indexing; -(_if this value is increasing rapidly indicates {es} backpressure_) +(_if this value is increasing rapidly it indicates {es} backpressure_) * `beats_stats.metrics.libbeat.output.events.acked` that represents the number of indexing operations that have completed successfully; * `beats_stats.metrics.libbeat.output.events.failed` that represents the number of indexing operations that failed, it includes all failures; -(_if this value is increasing rapidly indicates {es} backpressure_) +(_if this value is increasing rapidly it indicates {es} backpressure_) * `beats_stats.metrics.libbeat.output.events.toomany` that represents the number of indexing operations that failed due to {es} responding with 429 Too many Requests; -(_if this value is increasing rapidly indicates {es} backpressure_) +(_if this value is increasing rapidly it indicates {es} backpressure_) * `beats_stats.output.elasticsearch.bulk_requests.available` that represents the number of bulk indexers available for making bulk index requests; -(_if this value is equal to 0 indicates {es} backpressure_) +(_if this value is equal to 0 it indicates {es} backpressure_) * `beats_stats.output.elasticsearch.bulk_requests.completed` that represents the number of already completed bulk requests; * `beats_stats.metrics.output.elasticsearch.indexers.active` that represents the number of active bulk indexers that are concurrently processing batches; diff --git a/docs/en/observability/apm/troubleshoot-apm.asciidoc b/docs/en/observability/apm/troubleshoot-apm.asciidoc index ce51906aa2..4df6fdc24f 100644 --- a/docs/en/observability/apm/troubleshoot-apm.asciidoc +++ b/docs/en/observability/apm/troubleshoot-apm.asciidoc @@ -9,6 +9,7 @@ and processing and performance guidance. * <> * <> * <> +* <> For additional help with other APM components, see the links below. From 43be488f220f99a2262264b1527906cb79b45d39 Mon Sep 17 00:00:00 2001 From: Kostiantyn Masliuk <1pkg@protonmail.com> Date: Thu, 1 Aug 2024 12:11:52 -0700 Subject: [PATCH 6/8] address comments --- .../apm/apm-performance-diagnostic.asciidoc | 36 +++++++++++-------- 1 file changed, 22 insertions(+), 14 deletions(-) diff --git a/docs/en/observability/apm/apm-performance-diagnostic.asciidoc b/docs/en/observability/apm/apm-performance-diagnostic.asciidoc index caab1a04f7..b4b554d3c3 100644 --- a/docs/en/observability/apm/apm-performance-diagnostic.asciidoc +++ b/docs/en/observability/apm/apm-performance-diagnostic.asciidoc @@ -7,7 +7,7 @@ When {es} is under excessive load or indexing pressure, APM Server could experience the downstream backpressure when indexing new documents into {es}. Most commonly, backpressure from {es} will manifest itself in the form of higher indexing latency and/or rejected requests, which in return could lead APM Server to deny incoming requests. -As a result APM agents connected to the affected APM Server will suffer from throttling and/or request timeout when shipping APM events. +As a result, APM agents connected to the affected APM Server will suffer from throttling and/or request timeout when shipping APM events. To quickly identify possible issues try looking for similar error logs lines in APM Server logs: @@ -19,25 +19,33 @@ To quickly identify possible issues try looking for similar error logs lines in ... ---- -To gain better insight into APM Server health and performance, consider enabling the monitoring feature by following the steps in <>. -When enabled APM Server will additionally report a set of vital metrics to help you identify any performance degradation. +To gain better insight into APM Server health and performance, consider enabling the monitoring feature by following the steps in <>. +When enabled, APM Server will additionally report a set of vital metrics to help you identify any performance degradation. Pay careful attention to the next metric fields: -* `beats_stats.metrics.libbeat.output.events.active` that represents the number of buffered pending documents waiting for indexing; -(_if this value is increasing rapidly it indicates {es} backpressure_) -* `beats_stats.metrics.libbeat.output.events.acked` that represents the number of indexing operations that have completed successfully; -* `beats_stats.metrics.libbeat.output.events.failed` that represents the number of indexing operations that failed, it includes all failures; -(_if this value is increasing rapidly it indicates {es} backpressure_) -* `beats_stats.metrics.libbeat.output.events.toomany` that represents the number of indexing operations that failed due to {es} responding with 429 Too many Requests; -(_if this value is increasing rapidly it indicates {es} backpressure_) +* `beats_stats.metrics.libbeat.output.events.active` that represents the number of buffered pending documents waiting to be ingested; +(_if this value is increasing rapidly it may indicate {es} backpressure_) +* `beats_stats.metrics.libbeat.output.events.acked` that represents the total number of documents that have been ingested successfully; +* `beats_stats.metrics.libbeat.output.events.failed` that represents the total number of documents that failed to ingest; +(_if this value is increasing rapidly it may indicate {es} backpressure_) +* `beats_stats.metrics.libbeat.output.events.toomany` that represents the number of documents that failed to ingest due to {es} responding with 429 Too many Requests; +(_if this value is increasing rapidly it may indicate {es} backpressure_) * `beats_stats.output.elasticsearch.bulk_requests.available` that represents the number of bulk indexers available for making bulk index requests; -(_if this value is equal to 0 it indicates {es} backpressure_) +(_if this value is equal to 0 it may indicate {es} backpressure_) * `beats_stats.output.elasticsearch.bulk_requests.completed` that represents the number of already completed bulk requests; * `beats_stats.metrics.output.elasticsearch.indexers.active` that represents the number of active bulk indexers that are concurrently processing batches; -See https://www.elastic.co/guide/en/beats/metricbeat/current/exported-fields-beat.html[{metricbeat} documentation] for the full list of exported metric fields. +See {metricbeat-ref}/exported-fields-beat.html[{metricbeat} documentation] for the full list of exported metric fields. One likely cause of excessive indexing pressure or rejected requests is undersized {es}. To mitigate this, follow the guidance in {ref}/rejected-requests.html[Rejected requests]. -If scaling {es} resources up is not an option, you can try to workaround by adjusting `flush_bytes`, `flush_interval`, `max_retries` and `timeout` settings described in <> to reduce APM Server indexing pressure. -However, consider that increasing number of buffered documents and/or reducing retries may lead to a higher rate of dropped APM events. \ No newline at end of file + +If scaling {es} resources up is not an option, you can adjust the `flush_bytes`, `flush_interval`, `max_retries` and `timeout` settings described in <> to reduce APM Server indexing pressure. However, consider that increasing number of buffered documents and/or reducing retries may lead to a higher rate of dropped APM events. Down bellow a custom configuration example is listed where the number of default buffered documents is roughly doubled while {es} indexing retries are decreased simultaneously. This configuration provides a generic example and might not be applicable to your situation. Try adjusting the settings further to see what works for you. +[source,yaml] +---- +output.elasticsearch: + flush_bytes: "2MB" # double the default value + flush_interval: "2s" # double the default value + max_retries: 1 # decrease the default value + timeout: 60 # decrease the default value +---- \ No newline at end of file From f73d8a754a53a356be66defadaf8d52b8868795c Mon Sep 17 00:00:00 2001 From: Kostiantyn Masliuk <1pkg@protonmail.com> Date: Fri, 2 Aug 2024 12:00:51 -0700 Subject: [PATCH 7/8] fix doc typo --- docs/en/observability/apm/apm-performance-diagnostic.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/en/observability/apm/apm-performance-diagnostic.asciidoc b/docs/en/observability/apm/apm-performance-diagnostic.asciidoc index b4b554d3c3..4af3703264 100644 --- a/docs/en/observability/apm/apm-performance-diagnostic.asciidoc +++ b/docs/en/observability/apm/apm-performance-diagnostic.asciidoc @@ -40,7 +40,7 @@ See {metricbeat-ref}/exported-fields-beat.html[{metricbeat} documentation] for t One likely cause of excessive indexing pressure or rejected requests is undersized {es}. To mitigate this, follow the guidance in {ref}/rejected-requests.html[Rejected requests]. -If scaling {es} resources up is not an option, you can adjust the `flush_bytes`, `flush_interval`, `max_retries` and `timeout` settings described in <> to reduce APM Server indexing pressure. However, consider that increasing number of buffered documents and/or reducing retries may lead to a higher rate of dropped APM events. Down bellow a custom configuration example is listed where the number of default buffered documents is roughly doubled while {es} indexing retries are decreased simultaneously. This configuration provides a generic example and might not be applicable to your situation. Try adjusting the settings further to see what works for you. +If scaling {es} resources up is not an option, you can adjust the `flush_bytes`, `flush_interval`, `max_retries` and `timeout` settings described in <> to reduce APM Server indexing pressure. However, consider that increasing number of buffered documents and/or reducing retries may lead to a higher rate of dropped APM events. Down below a custom configuration example is listed where the number of default buffered documents is roughly doubled while {es} indexing retries are decreased simultaneously. This configuration provides a generic example and might not be applicable to your situation. Try adjusting the settings further to see what works for you. [source,yaml] ---- output.elasticsearch: From bc97ba661f404e73226f82d4aa1edeaa2dd107bf Mon Sep 17 00:00:00 2001 From: Kostiantyn Masliuk <1pkg@protonmail.com> Date: Mon, 5 Aug 2024 14:40:56 -0700 Subject: [PATCH 8/8] add not recommended banner --- docs/en/observability/apm/apm-performance-diagnostic.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/en/observability/apm/apm-performance-diagnostic.asciidoc b/docs/en/observability/apm/apm-performance-diagnostic.asciidoc index 4af3703264..daa4d6f12a 100644 --- a/docs/en/observability/apm/apm-performance-diagnostic.asciidoc +++ b/docs/en/observability/apm/apm-performance-diagnostic.asciidoc @@ -40,7 +40,7 @@ See {metricbeat-ref}/exported-fields-beat.html[{metricbeat} documentation] for t One likely cause of excessive indexing pressure or rejected requests is undersized {es}. To mitigate this, follow the guidance in {ref}/rejected-requests.html[Rejected requests]. -If scaling {es} resources up is not an option, you can adjust the `flush_bytes`, `flush_interval`, `max_retries` and `timeout` settings described in <> to reduce APM Server indexing pressure. However, consider that increasing number of buffered documents and/or reducing retries may lead to a higher rate of dropped APM events. Down below a custom configuration example is listed where the number of default buffered documents is roughly doubled while {es} indexing retries are decreased simultaneously. This configuration provides a generic example and might not be applicable to your situation. Try adjusting the settings further to see what works for you. +(Not recommended) If scaling {es} resources up is not an option, you can adjust the `flush_bytes`, `flush_interval`, `max_retries` and `timeout` settings described in <> to reduce APM Server indexing pressure. However, consider that increasing number of buffered documents and/or reducing retries may lead to a higher rate of dropped APM events. Down below a custom configuration example is listed where the number of default buffered documents is roughly doubled while {es} indexing retries are decreased simultaneously. This configuration provides a generic example and might not be applicable to your situation. Try adjusting the settings further to see what works for you. [source,yaml] ---- output.elasticsearch: