[Flaky Test]: TestRpmLogIngestFleetManaged/Monitoring_logs_are_shipped – failed to evaluate all symlinks

[rdner]

Failing test case

TestRpmLogIngestFleetManaged/Monitoring_logs_are_shipped

Error message

failed to evaluate all symlinks

Build

https://buildkite.com/elastic/elastic-agent-extended-testing/builds/625#0190110f-8604-4125-9789-621c5241ef2b

OS

Linux

Stacktrace and notes

=== RUN   TestRpmLogIngestFleetManaged/Monitoring_logs_are_shipped
    logs_ingestion_test.go:489: Making sure metricbeat logs are populated
    logs_ingestion_test.go:493: metricbeat: Got 300 documents
    logs_ingestion_test.go:498: Making sure all components are healthy
    logs_ingestion_test.go:500: 
        	Error Trace:	/home/rhel/agent/testing/integration/logs_ingestion_test.go:500
        	            				/home/rhel/agent/testing/integration/logs_ingestion_test.go:241
        	Error:      	Received unexpected error:
        	            	could not unmarshal agent status output: error: error creating cmd: failed to get control protcol address: failed to evaluate all symlinks of /tmp/TestRpmLogIngestFleetManaged3099371811/001/elastic-agent-8.15.0-SNAPSHOT-x86_64: lstat /tmp/TestRpmLogIngestFleetManaged3099371811/001/elastic-agent-8.15.0-SNAPSHOT-x86_64: no such file or directory, output: 
        	            	unexpected end of JSON input
        	Test:       	TestRpmLogIngestFleetManaged/Monitoring_logs_are_shipped
        	Messages:   	could not get agent status to verify all components are healthy
--- FAIL: TestRpmLogIngestFleetManaged/Monitoring_logs_are_shipped (15.30s)

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[rdner]

Also happened in:

• https://buildkite.com/elastic/elastic-agent-extended-testing/builds/626#0190110f-ba61-431a-9d55-4fc925cd5600
• https://buildkite.com/elastic/elastic-agent-extended-testing/builds/627#0190110f-b8c4-42ea-8b61-ad907bdf0f08
• https://buildkite.com/elastic/elastic-agent-extended-testing/builds/628#01901110-3058-4208-a2be-c3f1da97bdcc

[cmacknz]

The failing line is:

elastic-agent/pkg/testing/fixture.go

Lines 670 to 674 in e8962b0

	cAddr, err := control.AddressFromPath(f.operatingSystem, f.workDir)
	if err != nil {
	return nil, fmt.Errorf("failed to get control protcol address: %w", err)
	}
	agentClient := client.New(client.WithAddress(cAddr))

failed to evaluate all symlinks of /tmp/TestRpmLogIngestFleetManaged3099371811/001/elastic-agent-8.15.0-SNAPSHOT-x86_64: no such file or directory

The DEB and RPM tests are the ones that are failing, and this makes some sense, because they do not use any paths under /tmp and that won't be the path to the agent command. For those packages the symlink ends up under /var/lib.

elastic-agent/dev-tools/packaging/packages.yml

Line 66 in e8962b0

/var/lib/{{.BeatName}}/data/{{.BeatName}}-{{agent_package_version}}{{snapshot_suffix}}-{{ commit_short }}/{{.BeatName}}{{.BinaryExt}}:

I think the DEB and RPM install commands are missing setting up a client at the right socket path. I am actually surprised this failure doesn't happen every time more than anything.

Here is the client getting created on the regular install path:

elastic-agent/pkg/testing/fixture_install.go

Lines 205 to 217 in e8962b0

	// we just installed agent, the control socket is at a well-known location
	socketPath := fmt.Sprintf("unix://%s", paths.ControlSocketRunSymlink) // use symlink as that works for all versions
	if runtime.GOOS == "windows" {
	// Windows uses a fixed named pipe, that is always the same.
	// It is the same even running in unprivileged mode.
	socketPath = paths.WindowsControlSocketInstalledPath
	} else if !installOpts.Privileged {
	// Unprivileged versions move the socket to inside the installed directory
	// of the Elastic Agent.
	socketPath = paths.ControlSocketFromPath(runtime.GOOS, f.workDir)
	}
	c := client.New(client.WithAddress(socketPath))
	f.setClient(c)

Here is the DEB install which does not set this up, so we fall back to trying to find the control socket in work dir, which is a temporary directory.

elastic-agent/pkg/testing/fixture_install.go

Line 435 in e8962b0

enrollArgs := []string{"elastic-agent", "enroll"}

The DEB and RPM install are also missing the clean up calls to get diagnostics and dump processes.

elastic-agent/pkg/testing/fixture_install.go

Lines 251 to 254 in e8962b0

	f.t.Cleanup(func() {
	// check for running agents after uninstall had a chance to run
	assert.Empty(f.t, getElasticAgentProcesses(f.t), "there should be no running agent at the end of the test")
	})

[ycombinator]

Another failure in https://buildkite.com/elastic/elastic-agent-extended-testing/builds/1918#01915121-b869-4cc3-9ab1-fa053f0eb39f

[rdner]

@ycombinator the failure you linked seems to be new/unrelated, see #5311

[VihasMakwana]

This test is passing now, thanks to #4938.
Offending code.
Closing this.