Add the full set of ECS metadata sent to Fleet to Diagnostics

[cmacknz]

What does this PR do?

Adds the full set of ECS metadata that is sent to Fleet when the agent checks in to diagnostics. If you don't know what that is, it's this:

elastic-agent/internal/pkg/agent/application/gateway/fleet/fleet_gateway.go

Lines 327 to 330 in d2047ac

	ecsMeta, err := info.Metadata(ctx, f.log)
	if err != nil {
	f.log.Error(errors.New("failed to load metadata", err))
	}

elastic-agent/internal/pkg/agent/application/info/agent_metadata.go

Lines 25 to 31 in d2047ac

	// ECSMeta is a collection of agent related metadata in ECS compliant object form.
	type ECSMeta struct {
	Elastic *ElasticECSMeta `json:"elastic"`
	Host *HostECSMeta `json:"host"`
	OS *SystemECSMeta `json:"os"`
	}

Why is it important?

When all providers were enabled by default, you were able to get the information about the host from the host provider (hostname, OS version, etc) in the variables.yaml file. Now that providers are only enabled when referenced explicitly in the policy, the content of the variables.yaml file is by default:

variables:
    - {}

This makes sure that detailed information about the host is available in diagnostics all the time separately from providers. This is useful if you are say, debugging why you can't find agent by the hostname you think it should have in Fleet. The format of the agent-info.yaml file now looks like:

headers: {}
log_level: info
log_level_raw: info
metadata:
    elastic:
        agent:
            buildoriginal: '9.1.0-SNAPSHOT (build: d2047ac48df2f4536ca69a86ad4922b3e264501a at 2025-02-25 21:52:49 +0000 UTC)'
            complete: false
            id: 9ba45d59-e33f-42c7-a47d-50a9a37705ef
            loglevel: info
            snapshot: true
            unprivileged: false
            upgradeable: true
            version: 9.1.0
    host:
        arch: arm64
        hostname: Craigs-Macbook-Pro.local
        id: 48DA13D6-B83B-5C71-A4F3-494E674F9F37
        ip:
            - 127.0.0.1/8
            - ...
        mac:
            - ...
        name: craigs-macbook-pro.local
    os:
        family: darwin
        fullname: macOS(14.7.3)
        kernel: 23.6.0
        name: macOS
        platform: darwin
        version: 14.7.3

Disruptive User Impact

I removed some of the duplicated keys from the top level of agent-info.yaml. I am assuming nobody was relying on this in a critical way.

How to test this PR locally

elastic-development-agent diagnostics and then read the agent-info.yaml file.

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[mergify]

This pull request does not have a backport label. Could you fix it @cmacknz? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-./d./d is the label that automatically backports to the 8./d branch. /d is the digit
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

[elasticmachine]

💔 Build Failed

• Buildkite Build
• Commit: 08e75da

Failed CI Steps

• check-ci
• Unit tests - Ubuntu 22.04
• Unit tests - Ubuntu 22.04 ARM64
• Unit tests - Windows 2022
• Unit tests - Windows 2016
• Unit tests - macOS 15 ARM
• Unit tests - Windows 10
• Unit tests - Windows 11
• null

History

cc @cmacknz

[elastic-sonarqube]

Quality Gate passed

Issues
1 New issue
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
85.7% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[blakerouse]

Didn't think about how the host information would not be included in variables by default would affect the information provided in diagnostics.

Overall I am good with this change, but will wait for passing CI before giving approval.