[k8s] embed hints inputs in agent container image

[pkoutsovasilis]

What does this PR do?

This PR embeds the hints-based autodiscovery templates directly into the Elastic Agent container image. These templates, previously fetched by an init container, are now included in the image under /usr/share/elastic-agent/hints.inputs.d. The Elastic Agent has been updated to optionally use these embedded templates based on the Kubernetes hints configuration in the user's settings. If hints-based autodiscovery is enabled, the Elastic Agent dynamically reads from the embedded templates directory, ensuring seamless integration without external dependencies.

Additionally:

• Introduced a new mechanism for discovering hints-based inputs dynamically from the embedded directory.
• Updated CI to package the templates into the container image during the build process.
• Modified the Helm chart to remove the dependency on an init container for downloading the templates.

Why is it important?

This change improves the robustness and efficiency of Elastic Agent deployment in Kubernetes environments by:

• Eliminating the need for an init container to fetch and extract the templates during pod initialisation.
• Reducing potential errors due to external network issues or offline environments.
• Accelerating the pod startup process by preloading the templates into the container image.
• Supporting environments with strict network policies or without external internet access.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

Disruptive User Impact

This change should not cause any disruption for users. For users who enable hints-based autodiscovery, the behavior remains consistent, with improved reliability. For users who do not enable this feature, the embedded templates will remain unused, ensuring no impact on performance or functionality.

How to test this PR locally

mage integration:auth
PLATFORMS=linux/arm64 EXTERNAL=true SNAPSHOT=true PACKAGES=docker mage -v package 
INSTANCE_PROVISIONER=kind STACK_PROVISIONER=stateful K8S_VERSION=v1.31.1 SNAPSHOT=true mage integration:kubernetes

Related issues

• Closes Investigate the copy of autodiscovery templates inside Elastic Agent image #5661

[elastic-sonarqube]

Quality Gate passed

Issues
1 New issue
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
43.8% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[swiatekm]

LGTM, but I'm also not an expert on agent packaging, so more eyes on this would be welcome.

[pkoutsovasilis]

ty both @swiatekm and @andrzej-stencel for the reviews, since you didn't spot anything major I feel we are on the right track 🙂 That said, I am gonna wait a little bit more, hopefully for a review from @blakerouse and @gizas just to make sure that there isn't something that I missed since I am relatively new to this parts of code

[blakerouse]

Looks good.

Determining if providers.kubernetes.hints.enabled didn't turn out as bad as I thought it was going to.