From 8c4c64d138ec45451586baceaf8a1ff76947b923 Mon Sep 17 00:00:00 2001 From: "mergify[bot]" <37929162+mergify[bot]@users.noreply.github.com> Date: Tue, 6 Aug 2024 09:54:29 -0700 Subject: [PATCH] [8.15](backport #4097) Add doc for diagnosing backpressure from Elasticsearch (#4122) * Add doc for diagnosing backpressure from Elasticsearch (#4097) * initial apm-es-backpressure doc draft * address review comments * fix internal doc references * address review comments * es backpressure troubleshoot doc fmt fix * address comments * fix doc typo * add not recommended banner (cherry picked from commit 406f6762881c1afaa619270ac15b60c23dcef2a1) * Delete docs/en/serverless directory --------- Co-authored-by: Kostiantyn Masliuk <1pkg@protonmail.com> Co-authored-by: github-actions[bot] --- .../apm/apm-performance-diagnostic.asciidoc | 51 +++++++++++++++++++ .../apm/troubleshoot-apm.asciidoc | 5 +- 2 files changed, 55 insertions(+), 1 deletion(-) create mode 100644 docs/en/observability/apm/apm-performance-diagnostic.asciidoc diff --git a/docs/en/observability/apm/apm-performance-diagnostic.asciidoc b/docs/en/observability/apm/apm-performance-diagnostic.asciidoc new file mode 100644 index 0000000000..daa4d6f12a --- /dev/null +++ b/docs/en/observability/apm/apm-performance-diagnostic.asciidoc @@ -0,0 +1,51 @@ +[[apm-performance-diagnostic]] +=== APM Server performance diagnostic + +[[apm-es-backpressure]] +[float] +==== Diagnosing backpressure from {es} + +When {es} is under excessive load or indexing pressure, APM Server could experience the downstream backpressure when indexing new documents into {es}. +Most commonly, backpressure from {es} will manifest itself in the form of higher indexing latency and/or rejected requests, which in return could lead APM Server to deny incoming requests. +As a result, APM agents connected to the affected APM Server will suffer from throttling and/or request timeout when shipping APM events. + +To quickly identify possible issues try looking for similar error logs lines in APM Server logs: + +[source,json] +---- +... +{"log.level":"error","@timestamp":"2024-07-27T23:46:28.529Z","log.origin":{"function":"github.com/elastic/go-docappender/v2.(*Appender).flush","file.name":"v2@v2.2.0/appender.go","file.line":370},"message":"bulk indexing request failed","service.name":"apm-server","error":{"message":"flush failed (429): [429 Too Many Requests]"},"ecs.version":"1.6.0"} +{"log.level":"error","@timestamp":"2024-07-27T23:55:38.612Z","log.origin":{"function":"github.com/elastic/go-docappender/v2.(*Appender).flush","file.name":"v2@v2.2.0/appender.go","file.line":370},"message":"bulk indexing request failed","service.name":"apm-server","error":{"message":"flush failed (503): [503 Service Unavailable]"},"ecs.version":"1.6.0"} +... +---- + +To gain better insight into APM Server health and performance, consider enabling the monitoring feature by following the steps in <>. +When enabled, APM Server will additionally report a set of vital metrics to help you identify any performance degradation. + +Pay careful attention to the next metric fields: + +* `beats_stats.metrics.libbeat.output.events.active` that represents the number of buffered pending documents waiting to be ingested; +(_if this value is increasing rapidly it may indicate {es} backpressure_) +* `beats_stats.metrics.libbeat.output.events.acked` that represents the total number of documents that have been ingested successfully; +* `beats_stats.metrics.libbeat.output.events.failed` that represents the total number of documents that failed to ingest; +(_if this value is increasing rapidly it may indicate {es} backpressure_) +* `beats_stats.metrics.libbeat.output.events.toomany` that represents the number of documents that failed to ingest due to {es} responding with 429 Too many Requests; +(_if this value is increasing rapidly it may indicate {es} backpressure_) +* `beats_stats.output.elasticsearch.bulk_requests.available` that represents the number of bulk indexers available for making bulk index requests; +(_if this value is equal to 0 it may indicate {es} backpressure_) +* `beats_stats.output.elasticsearch.bulk_requests.completed` that represents the number of already completed bulk requests; +* `beats_stats.metrics.output.elasticsearch.indexers.active` that represents the number of active bulk indexers that are concurrently processing batches; + +See {metricbeat-ref}/exported-fields-beat.html[{metricbeat} documentation] for the full list of exported metric fields. + +One likely cause of excessive indexing pressure or rejected requests is undersized {es}. To mitigate this, follow the guidance in {ref}/rejected-requests.html[Rejected requests]. + +(Not recommended) If scaling {es} resources up is not an option, you can adjust the `flush_bytes`, `flush_interval`, `max_retries` and `timeout` settings described in <> to reduce APM Server indexing pressure. However, consider that increasing number of buffered documents and/or reducing retries may lead to a higher rate of dropped APM events. Down below a custom configuration example is listed where the number of default buffered documents is roughly doubled while {es} indexing retries are decreased simultaneously. This configuration provides a generic example and might not be applicable to your situation. Try adjusting the settings further to see what works for you. +[source,yaml] +---- +output.elasticsearch: + flush_bytes: "2MB" # double the default value + flush_interval: "2s" # double the default value + max_retries: 1 # decrease the default value + timeout: 60 # decrease the default value +---- \ No newline at end of file diff --git a/docs/en/observability/apm/troubleshoot-apm.asciidoc b/docs/en/observability/apm/troubleshoot-apm.asciidoc index 48af1af1c1..5823d9c248 100644 --- a/docs/en/observability/apm/troubleshoot-apm.asciidoc +++ b/docs/en/observability/apm/troubleshoot-apm.asciidoc @@ -9,6 +9,7 @@ and processing and performance guidance. * <> * <> * <> +* <> For additional help with other APM components, see the links below. @@ -54,4 +55,6 @@ include::apm-response-codes.asciidoc[] include::processing-performance.asciidoc[] -include::{observability-docs-root}/docs/en/observability/apm/debugging.asciidoc[] \ No newline at end of file +include::{observability-docs-root}/docs/en/observability/apm/debugging.asciidoc[] + +include::apm-performance-diagnostic.asciidoc[] \ No newline at end of file