From 016d36c7ce4254f0e1dbc6cf03dcf349a3735a67 Mon Sep 17 00:00:00 2001 From: Carson Ip Date: Thu, 18 Jul 2024 18:34:48 +0100 Subject: [PATCH] APM known issues: Too many small bulk requests (#4047) * APM known issues: Too many small bulk requests * Add log * Add elastic cloud * Pencil in 8.14.3 (cherry picked from commit 0335c1674564568371c5c6a8dfed08a3ce241fe5) --- .../observability/apm/known-issues.asciidoc | 39 +++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/docs/en/observability/apm/known-issues.asciidoc b/docs/en/observability/apm/known-issues.asciidoc index c7b0937ebf..0b87b2df0f 100644 --- a/docs/en/observability/apm/known-issues.asciidoc +++ b/docs/en/observability/apm/known-issues.asciidoc @@ -258,3 +258,42 @@ Mappings update for metrics-apm.service_destination.10m-default failed due to Re // Link to fix A fix was released in 8.12.2: https://github.com/elastic/apm-server/pull/12219[elastic/apm-server#12219]. + + +*Performance regression: APM issues too many small bulk requests for Elasticsearch output* + +_APM Server versions: >=8.13.0, \<= 8.14.2_ + + +// Describe the conditions in which this issue occurs +If you're on APM server version >=8.13.0, \<= 8.14.2_, using Elasticsearch output, +do not specify any `output.elasticsearch.flush_bytes`, +and do not disable compression explicitly by setting `output.elasticsearch.compression_level` to `0`, +// Describe the behavior of the issue +APM server will issue smaller bulk requests of 24KB size, +and more bulk requests will need to be made to maintain the original throughput. +This causes Elasticsearch to experience higher load, +and APM server may exhibit Elasticsearch backpressure symptoms. + +// Describe why it happens +This happens because a performance regression was introduced, such that the default value of bulk indexer flush bytes +was reduced from 1MB to 24KB. + +Affected APM servers will emit the following log: + +[source,txt] +---- +flush_bytes config value is too small (0) and might be ignored by the indexer, increasing value to 24576 +---- + +To workaround the issue, modify the Elasticsearch output configuration in APM. + +* For APM Server binary +** In `apm-server.yml`, set `output.elasticsearch.flush_bytes: 1mib` +* For Fleet-managed APM (non-Elastic Cloud) +** In Fleet, open the Settings tab. +** Under Outputs, identify the Elasticsearch output that receives from APM, select the edit icon. +** In the Edit output flyout, in "Advanced YAML configuration" field, add line `flush_bytes: 1mib`. +* For Elastic Cloud +** It is not possible to edit the Fleet "Elastic Cloud internal output". + +// Link to fix +A fix will be released in 8.14.3: https://github.com/elastic/apm-server/pull/13576[elastic/apm-server#13576].