[filebeat] Introduce otel mode

[khushijain21]

To note

• This PR uses converters for config translation. This will help reuse and maintain translation logic across beats.
• All common "beats to OTel" logic is under libbeat/otelcommon/
• Output specific config translation logic is under libbeat/outputs/${output_type}
• It throws error if any output other than elasticsearch is configured
• It throws error when unsupported elasticsearch paramaters are configured (some of them are just documented tho)

Elasticsearch paramaters that are currently unsupported

indices
pipelines
parameters
preset
setup.ilm.* -> supported but the logic is not in place yet
proxy_disable -> supported but the logic is not in place yet
proxy_headers
compression_level
loadbalance
non_indexable_policy
allow_older_versions
escape_html
kerberos-*
max_retries

and some TLS paramaters that are currently not supported

// currently unsupported parameters
// ssl.curve_types
// ssl.ca_sha256
// ssl.ca_trustred_fingerprint
// ssl.renegotiation
// ssl.verification_mode:
// ssl.supported_protocols -> partially supported
// ssl.restart_on_cert_change.*
// ssl.client_authentication

The change in filebeat and agentbeat binary sizes before and after this changeset is as follows

// For filebeat before
➜  golang-crossbuild git:(main) ✗ du -h filebeat-darwin-arm64
141M    filebeat-darwin-arm64

➜  golang-crossbuild git:(fosb1) ✗ du -h filebeat-darwin-arm64
158M    filebeat-darwin-arm64

// In Agentbeat

➜  golang-crossbuild git:(main) ✗ du -h agentbeat-darwin-arm64
222M    agentbeat-darwin-arm64

➜  golang-crossbuild git:(main) ✗ du -h agentbeat-darwin-arm64
242M    agentbeat-darwin-arm64

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

cd x-pack/filebeat
mage build
./filebeat otel -c ./cmd/filebeat-otel.yml

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @khushijain21? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit

[mergify]

backport-8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label and remove the backport-8.x label.

[VihasMakwana]

Overall looks good. Just need to address lint failures and handle errors.

[ishleenk17]

@elastic/obs-infraobs-integrations can someone from your team have a look at the failing test case? Looks related to ceph module

ERROR: for metricbeat_9_0_0_1947305a16-snapshot_ceph_1 Cannot create container for service ceph: Conflict. The container name "/metricbeat_9_0_0_1947305a16-snapshot_ceph_1" is already in use by container "e9fe0491d7074767f28d7cd8259c1a9ce828de82a6279f00bfb1d25d2c33d14f". You have to remove (or rename) that container to be able to reuse that name.

cc: @lalit-satapathy

This seems to be a flaky run. Integration test is trying to use a container which was not yet deleted completely in the previous try. I have given it a rerun

So, it was indeed a flaky test. CI is passing now.

[VihasMakwana]

@elastic/obs-infraobs-integrations can someone from your team have a look at the failing test case? Looks related to ceph module

ERROR: for metricbeat_9_0_0_1947305a16-snapshot_ceph_1 Cannot create container for service ceph: Conflict. The container name "/metricbeat_9_0_0_1947305a16-snapshot_ceph_1" is already in use by container "e9fe0491d7074767f28d7cd8259c1a9ce828de82a6279f00bfb1d25d2c33d14f". You have to remove (or rename) that container to be able to reuse that name.

cc: @lalit-satapathy

This seems to be a flaky run. Integration test is trying to use a container which was not yet deleted completely in the previous try. I have given it a rerun

So, it was indeed a flaky test. CI is passing now.

Thank you for taking a look!

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b fosb1 upstream/fosb1
git merge upstream/main
git push upstream fosb1

[cmacknz]

How much larger does including this make the Filebeat and Agentbeat binaries on disk?

[khushijain21]

Before and After including this changeset in filebeat

➜  golang-crossbuild git:(main) ✗ du -h filebeat-darwin-arm64
141M    filebeat-darwin-arm64

➜  golang-crossbuild git:(fosb1) ✗ du -h filebeat-darwin-arm64
158M    filebeat-darwin-arm64

In Agentbeat

➜  golang-crossbuild git:(main) ✗ du -h agentbeat-darwin-arm64
222M    agentbeat-darwin-arm64

➜  golang-crossbuild git:(main) ✗ du -h agentbeat-darwin-arm64
242M    agentbeat-darwin-arm64

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b fosb1 upstream/fosb1
git merge upstream/main
git push upstream fosb1

[leehinman]

Overall looking good.

Couple questions below.

One Nit, can we avoid "otelcommon" as a package name?

[leehinman]

As a follow on, I'd like to see an integration test that runs filebeat with a config that writes an event to elasticsearch, then run the same config with the otel command to prove that the converted config functions as intended. Test could use benchmark input and mock-es for the elasticsearch.

[rdner]

@leehinman the tests you described are a part of the follow up ticket here https://github.com/elastic/ingest-dev/issues/4595

Feel free to comment on the ticket if we missed something.