Auditbeat,Metricbeat - add /inputs/ to HTTP monitoring endpoint

[andrewkroh]

Proposed commit message

Make metrics published by "inputs" available through the /inputs/ route on the HTTP monitoring endpoint of Auditbeat and Metricbeat.

For Agent, include a snapshot of those metrics within the Agent diagnostics bundle as "input_metrics.json".

When running under Agent, each module instance is configured with only a single metricset. That module is given a unique id. That ID is what will be used as the id within the /inputs/ data. And that id will also be added as context to the logger that is passed into every metricset so that any log messages from a metricset can be associated back to the agent stream ID).

Relates: #36945

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

• Relates Add input metrics to Auditbeat #36945

Example

AFAICT no MetricSet implementations in Metricbeat or Auditbeat register any metrics so for now you can only use this to see what instances are running.

curl "http://localhost:6060/inputs/?pretty"

[
  {
    "ephemeral_id": "86b09717-49ac-4c66-b08c-75edc3b4cffc",
    "id": "audit/system-system_audit.package-1af2373c-24b7-43b2-a9f2-8f04f803a0c6",
    "input": "system/package",
    "starttime": "2023-10-26T18:11:10.850Z"
  },
  {
    "ephemeral_id": "52965779-c346-4061-9fac-0baea45d2461",
    "id": "audit/file_integrity-fim.event-7580a9e4-e355-4a26-95c8-e49f08bd4fd0",
    "input": "file_integrity/file",
    "starttime": "2023-10-26T18:11:10.850Z"
  }
]

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @andrewkroh? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Duration: 68 min 27 sec

❕ Flaky test report

No test was executed to be analysed.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[kunisen]

Hi team, it seems our doc is not accurate enough:
https://www.elastic.co/guide/en/beats/packetbeat/current/protocol-metrics-packetbeat.html

These metrics are exposed under the /inputs path.

It says /inputs but actually it should be /inputs/ (with a trailing /).
If we don't specify that trailing slash, we get nothing.

We think it's a doc bug, and can we have someone from beats team to help us fix this? 🙏
Please let us know if the above understanding is wrong and thanks!

cc @jasonyoum