WIP: otelcol: use ocb to autogenerate source files

[mauri870]

What does this PR do?

This PR uses the OpenTelemetry collector builder (ocb for short) to automate the generation of collector source files.

Why is it important?

Updating otel collector files manually can be time consuming and error prone, we can let ocb generate the source files.

[mergify]

This pull request does not have a backport label. Could you fix it @mauri870? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v./d./d./d is the label to automatically backport to the 8./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

[mauri870]

Unfortunately, we can't manage versions in this file and sync them with a go.mod file, since the dependencies are defined in the elastic-agent go.mod located at the root of the repository. We could consider using Go workspaces and moving this code to ${REPO_ROOT}/otel, for example, so it can be its own module with its own go.mod file. This approach would also clean up the otelcol dependencies in the root go.mod file.

[ycombinator]

We have dependabot currently bumping up the OTel dependency versions in go.mod: https://github.com/elastic/elastic-agent/blob/main/.github/dependabot.yml. Perhaps we can add a new step or job to https://github.com/elastic/elastic-agent/blob/main/.github/workflows/post-dependabot.yml to do some parsing of the go.mod file and sync the versions in this file with the parsed results?

[mauri870]

This is a bit hacky since ocb expects to build a final binary. Having an otel collector embedded into another binary seems to be quite useful, I wonder if the folks at OpenTelemetry would be willing to support a ocb mode to build a library instead of a final binary.

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
96.8% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[michalpristas]

Make sure you include this in mage update target at root.
It would be nice if mage update could regenerate sources. Update go mod versions and then update readme file as it does today.

[andrzej-stencel]

In general, I like the idea of keeping OTel dependencies in a canonical Builder config Yaml file. For engineers oriented around the OpenTelemetry Collector, this is the idiomatic way to understand what components are included.

However, given that the Elastic Agent is not simply an OTel Collector binary built by the OTel Collector Builder, but rather a binary that incorporates OTel Collector libraries into an existing code base, and given the current feature set of the Builder, my opinion is that the benefit of declaring OTel dependencies in Builder config file does not outweigh the added complexity.

Introducing this approach adds an additional level of abstraction, which affects all engineers working on the Elastic Agent code (Elasticians or not), some of them possibly not knowing anything about OTel Collector or the OTel Collector Builder. It creates new problems like "do we maintain a separate go.mod file for the OTel dependencies" or "how do we keep the OTel dependencies up to date".

The big advantage of the current approach is its simplicity. It is based on regular Go imports and nothing else. This makes it simpler to integrate with tools like Dependabot for dependency upgrades.

My recommendation is to leave the current approach unchanged for now for the sake of simplicity and revisit this decision when circumstances change.

[ycombinator]

Thanks for weighing in, @andrzej-stencel. You raise a good point about the added complexity and additional level of abstraction that may not be obvious to all contributors. Unless others have strong opinions for favoring the approach in this PR, I'm OK with sticking with our current approach which is not as declarative as the one in this PR but is simpler to understand.

[mauri870]

Just an update: I ended up opening an issue upstream to suggest adding support for OCB library code generation. I don't expect this to be considered before the 1.0 release of the builder, but it is a valid feature request.

The changes proposed here aren't worth implementing, as they add unnecessary complexity compared to the current approach and require workarounds to bypass OCB's limitations.

With that in mind, I’m closing this for now.