monitoring: apm-server only ships a subset of monitoring metrics

[carsonip]

Discovered by #13244

EA managed apm-server is only shipping a subset of apm-server monitoring metrics. This limits observability of EA managed apm-server. The problem is spread over the following use cases:

• APM server self-monitoring
  • metricsets state, stats
  • indices (not data stream) .monitoring-beats-7-*
  • does NOT use TSDS, but fields are unmapped due to dynamic: false
  • code in ES repo monitoring-beats.json
• Stack monitoring: metricbeat beat-xpack module uses index template embedded in ES
  • metricsets state, stats
  • data stream .monitoring-beats-8-mb
  • use case: Elastic Cloud self-monitoring, metricbeat standalone beat-xpack
  • does NOT use TSDS, but fields are unmapped due to dynamic: false
  • code in ES repo monitoring-beats-mb.json
• Metricbeat without xpack: metricbeat beat module
  • metricsets state, stats
  • data stream metricbeat-*
  • use case: metricbeat standalone
  • does NOT use TSDS, fields are mapped dynamically
  • code in beats repo
• EA integration
  • metricsets stats only
  • data stream metrics-elastic_agent.apm_server.* (see code)
  • use case: elastic-agent standalone with monitoring
  • to see EA integration package content: POST kbn:/api/fleet/epm/packages/elastic_agent
  • uses TSDS, see [Elastic Agent] Enable TSDB by default integrations#7214 . The issue with using TSDS is that unmapped fields are dropped, as it uses synthetic source.
  • code in integrations repo
  • it doesn't contain any apm-server fields as of today

Unknowns:

• As the outdated mappings above contain fields that are no longer used, mapping generated from apm-server:5066/stats will not be pure addition. Not sure if dropping a field mapping will be problematic.
  • Should it be a superset of all historically available monitoring fields? That doesn't sound realistic either

[carsonip]

Testing

Completed local testing on all elastic/beats#40127, elastic/elasticsearch#110568, and elastic/integrations#10414. Note that both elastic/elasticsearch#110568 and elastic/integrations#10414 require elastic/beats#40127 to have output.elasticsearch.* metrics parsed.

apm-server self monitoring (indices .monitoring-beats-7-*)

before

after

metricbeat beat-xpack (DS .monitoring-beats-8-mb)

before

after

metricbeat beat standalone (DS metricbeat-*)

before

after

EA agent monitoring (DS metrics-elastic_agent-apm_server-*)

before

after

Long term solution

While the Python script in #13638 generates the correct mapping for all the above use cases, this approach is not very maintainable. Ideally, having dynamic: true like metricbeat standalone enables apm-server monitoring fields to be mapped dynamically and will require minimal maintenance. However, since EA agent mapping already has TSDB enabled, disabling TSDB now and enabling dynamic mapping sounds like a step back.

[lahsivjar]

Tested on 8.15 BC2, (UPDATE) followup testing with 8.15 BC3:

• APM server self-monitoring: tested with ESS elasticsearch, ❌ the backport PR was not merged. It is merged now, and will require testing again in BC3. Tested with updated BC and is working as expected.

  Snapshots from testing

  

• APM server stack monitoring: tested on ESS, ❌ the backport PR was not merged. It is merged now, and will require testing again in BC3. Tested with updated BC and is working as expected.

  Snapshots from testing

  

• Metricbeat without x-pack, tested locally with apm-server 008b12a7e9929ce00e01040e67e83cf19c0b7de7 and beats 417ac43a6193bdf26da490d41532b2f2d5bc3f70

  Snapshots from testing

  

• EA integration tested with locally running EA using elastic-agent-package 475d247d16a94af14b75d991a8da3d63d68e8042. ❌ While the test succeeded, I had to manually upgrade EA integration as the version bundled in Kibana is still older (v1.20.0) whereas the latest version is v2.0.1 - since this is not related to the current issue, marking this point as done. Confirmed that with BC3 the latest version v2.0.1 of elastic-agent integration is bundled.

  Snapshots from testing

  

[carsonip]

Moved all remaining tasks to #13731 . Closing this issue as the bug of missing metrics is now fixed.