integration deb rpm

[leehinman]

What does this PR do?

Add an integration test that installs elastic-agent via the DEB package and makes sure that logs are sent to Elasticsearch.
This is a retry of #4301 with what should be a fix for the serverless builds.

Why is it important?

Need to test that DEB packages work.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
  - [ ] I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

Author's Checklist

• [ ]

How to test this PR locally

mage integration:single TestDebLogIngestFleetManaged

Related issues

• Relates The integration test framework should test the DEB and RPM installations of the agent #4086
• Relates add a DEB integration test #4301

Use cases

Screenshots

Logs

Questions to ask yourself

• How are we going to support this in production?
• How are we going to measure its adoption?
• How are we going to debug this?
• What are the metrics I should take care of?
• ...

[mergify]

This pull request does not have a backport label. Could you fix it @leehinman? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v./d./d./d is the label to automatically backport to the 8./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

[rdner]

Note: Needs to be tested with Serverless Beats Tests before merged.

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[leehinman]

.buildkit/scripts/steps/beats_tests.sh works now.

elastic2:elastic-agent[4086_integration_deb_rpm]% ./.buildkite/scripts/steps/beats_tests.sh
~/src/elastic-agent ~/src/elastic-agent
go version go1.21.8 darwin/arm64
~/src/elastic-agent
~/src/elastic-agent ~/src/elastic-agent
Targets:
  addLicenseHeaders                   adds license headers
  assembleDarwinUniversal             merges the darwin/amd64 and darwin/arm64 into a single universal binary using `lipo`.
  build:all*                          build all the things for the current projects.
  build:binary                        build the fleet artifact.
  build:binaryOSS                     build the fleet artifact.
  build:clean                         up dev environment.
  build:generateConfig                generates the configuration from _meta/elastic-agent.yml
  build:testBinaries                  build the required binaries for the test suite.
  buildFleetCfg                       embed the default fleet configuration as part of the binary.
  buildGoDaemon                       builds the go-daemon binary (use crossBuildGoDaemon).
  buildPGP
  check                               formats code, updates generated content, check for common errors, and checks for any modified files.
  check:all                           run all the code checks.
  check:changes                       run git status --porcelain and return an error if we have changes or uncommitted files.
  check:goLint                        run the code through the linter.
  check:license                       makes sure that all the Golang files have the appropriate license header.
  checkLicenseHeaders                 checks license headers
  checkNoChanges
  clean                               cleans all generated files and build artifacts.
  cloud:image                         builds a cloud image
  cloud:push                          builds a cloud image tags it correctly and pushes to remote image repo.
  config                              generates both the short/reference/docker.
  controlProto                        generates pkg/agent/control/proto module.
  crossBuild                          cross-builds the beat for all target platforms.
  crossBuildGoDaemon                  cross-builds the go-daemon binary using Docker.
  demo:enroll                         runs agent which enrolls before running.
  demo:noEnroll                       runs agent which does not enroll before running.
  dev:build                           builds the agent binary with DEV flag set.
  dev:package                         bundles the agent binary with DEV flag set.
  downloadManifest                    downloads the provided manifest file into the predefined folder
  dumpVariables                       writes the template variables and values to stdout.
  fakeShipperProto                    generates pkg/component/fake/common event protocol.
  fixDRADockerArtifacts               is a workaround for the DRA artifacts produced by the package target.
  fmt                                 formats source code (.go and .py) and adds license headers.
  format:all                          format automatically all the codes.
  format:license                      applies the right license header.
  goGet                               fetch a remote dependencies.
  goIntegTest                         method informs that no integration tests will be executed.
  golangCrossBuild                    build the Beat binary inside of the golang-builder.
  golangCrossBuildOSS                 build the Beat binary inside of the golang-builder.
  integTest                           executes integration tests (it uses Docker to run the tests).
  integration:auth                    authenticates users who run it to various IaaS CSPs and ESS
  integration:check                   checks that integration tests are using define.Require
  integration:clean                   cleans up the integration testing leftovers
  integration:deployDebugTools        installs all necessary tools to debug tests from a VM
  integration:deployEnvFile           generates and deploys to a VM 'env.sh' containing envvars to connect to a cloud stack
  integration:generateEnvFile         generates 'env.sh' containing envvars to connect to a cloud stack
  integration:listInstances           lists all VMs in a human readable form, including connection details
  integration:local                   runs only the integration tests that support local mode it takes as argument the test name to run or all if we want to run them all.
  integration:matrix                  runs integration tests on a matrix of all supported remote hosts
  integration:prepareOnRemote         shouldn't be called locally (called on remote host to prepare it for testing)
  integration:printState              prints details about cloud stacks and VMs
  integration:single                  runs single integration test on remote host
  integration:ssh                     prints to stdout the SSH command to connect to a VM, a menu is printed to stderr.
  integration:stacks                  lists all stack deployments in a human readable form
  integration:test                    runs integration tests on remote hosts
  integration:testBeatServerless      Run beat serverless tests
  integration:testForResourceLeaks
  integration:testOnRemote            shouldn't be called locally (called on remote host to perform testing)
  integration:updateVersions          runs an update on the `.agent-versions.json` fetching the latest version list from the artifact API.
  ironbank                            Package packages elastic-agent for the IronBank distribution, relying on the binaries having already been built.
  notice                              Generates NOTICE.txt.
  otel:readme
  package                             packages the Beat for distribution.
  packageAgentCore                    cross-builds and packages distribution artifacts containing only elastic-agent binaries with no extra files or dependencies.
  packageSystemTests                  packages the python system tests results
  prepare:env                         returns information about the environment.
  prepare:installGoLicenser           install go-licenser to check license of the files.
  prepare:installGoLint               for the code.
  test                                runs all available tests (unitTest + integTest).
  test:all                            runs all the tests.
  test:coverage                       takes the coverages report from running all the tests and display the results in the browser.
  test:unit                           runs all the unit tests.
  testPackages                        tests the generated packages (i.e.
  unitTest                            performs unit test on agent.
  update                              is an alias for executing control protocol, configs, and specs.

* default target
~/src/elastic-agent
/tmp/beats-build ~/src/elastic-agent
Cloning into 'beats'...
remote: Enumerating objects: 16655, done.
remote: Counting objects: 100% (16655/16655), done.
remote: Compressing objects: 100% (10991/10991), done.
remote: Total 16655 (delta 5906), reused 11678 (delta 4635), pack-reused 0
Receiving objects: 100% (16655/16655), 106.11 MiB | 1.00 MiB/s, done.
Resolving deltas: 100% (5906/5906), done.
Updating files: 100% (13570/13570), done.
~/src/elastic-agent
testing metricbeat...
/tmp/beats-build/beats/x-pack/metricbeat ~/src/elastic-agent
hinman
total 320
drwxr-xr-x  18 hinman  wheel     576 Mar 19 13:13 .
drwxr-xr-x  14 hinman  wheel     448 Mar 19 13:13 ..
-rw-r--r--   1 hinman  wheel    6028 Mar 19 13:13 Jenkinsfile.yml
-rw-r--r--   1 hinman  wheel     196 Mar 19 13:13 Makefile
drwxr-xr-x   4 hinman  wheel     128 Mar 19 13:13 cmd
-rw-r--r--   1 hinman  wheel     274 Mar 19 13:13 conftest.py
-rw-r--r--   1 hinman  wheel     966 Mar 19 13:13 docker-compose.yml
drwxr-xr-x   3 hinman  wheel      96 Mar 19 13:13 include
-rw-r--r--   1 hinman  wheel    8260 Mar 19 13:13 magefile.go
-rw-r--r--   1 hinman  wheel     702 Mar 19 13:13 main.go
-rw-r--r--   1 hinman  wheel    1006 Mar 19 13:13 main_test.go
-rw-r--r--   1 hinman  wheel     314 Mar 19 13:13 metricbeat.docker.yml
-rw-r--r--   1 hinman  wheel  109068 Mar 19 13:13 metricbeat.reference.yml
-rw-r--r--   1 hinman  wheel    7011 Mar 19 13:13 metricbeat.yml
drwxr-xr-x  25 hinman  wheel     800 Mar 19 13:13 module
drwxr-xr-x  26 hinman  wheel     832 Mar 19 13:13 modules.d
drwxr-xr-x   3 hinman  wheel      96 Mar 19 13:13 scripts
drwxr-xr-x   3 hinman  wheel      96 Mar 19 13:13 tests
/tmp/beats-build/beats/x-pack/metricbeat /tmp/beats-build/beats/x-pack/metricbeat ~/src/elastic-agent
go version go1.21.8 darwin/arm64
/tmp/beats-build/beats/x-pack/metricbeat ~/src/elastic-agent
/tmp/beats-build/beats/x-pack/metricbeat /tmp/beats-build/beats/x-pack/metricbeat ~/src/elastic-agent
Generated fields.yml for metricbeat to /tmp/beats-build/beats/x-pack/metricbeat/fields.yml
>> Building metricbeat.yml for linux/amd64
>> Building metricbeat.reference.yml for linux/amd64
>> Building metricbeat.docker.yml for linux/amd64
>> buildGoDaemon: Building for linux/amd64
>> golangCrossBuild: Building for linux/amd64
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
>> Building using: cmd='build/mage-linux-arm64 buildGoDaemon', env=[CC=gcc, CXX=g++, GOARCH=amd64, GOARM=, GOOS=linux, PLATFORM_ID=linux-amd64]
>> Building using: cmd='build/mage-linux-arm64 golangCrossBuild', env=[CC=gcc, CXX=g++, GOARCH=amd64, GOARM=, GOOS=linux, PLATFORM_ID=linux-amd64]
/usr/bin/ld: /tmp/ccY3qE6k.o: in function `main':
god.c:(.text+0x2cd): warning: Using 'getgrnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/usr/bin/ld: god.c:(.text+0x263): warning: Using 'getpwnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
>> package: Building metricbeat type=tar.gz for platform=linux/amd64
>> Testing package contents
package ran for 4m39.582579041s
/tmp/beats-build/beats/x-pack/metricbeat ~/src/elastic-agent
~/src/elastic-agent
~/src/elastic-agent ~/src/elastic-agent
go version go1.21.8 darwin/arm64
~/src/elastic-agent
~/src/elastic-agent ~/src/elastic-agent
>>>> Using ogc instance provisioner
>>>> Using serverless stack provisioner
>>> Creating zip archive of repo to send to remote hosts
>>> Pulling latest ogc image
>>> Creating serverless stack 8.14.0-SNAPSHOT [stack_id: 8140-SNAPSHOT]
>>> Import layouts into ogc
>>> Created serverless stack 8.14.0-SNAPSHOT [stack_id: 8140-SNAPSHOT, deployment_id: c1c61c67869f464e95c718118e150d54]
>>> Waiting for serverless stack 8.14.0-SNAPSHOT to be ready [stack_id: 8140-SNAPSHOT, deployment_id: c1c61c67869f464e95c718118e150d54]
>>> Bring up instances through ogc
>>> Endpoints available: ES: https://snapshot-c1c61c.es.us-east-1.aws.elastic.cloud Fleet: https://snapshot-c1c61c.es.us-east-1.aws.elastic.cloud Kibana: https://snapshot-c1c61c.kb.us-east-1.aws.elastic.cloud
>>> Endpoints available: ES: https://snapshot-c1c61c.es.us-east-1.aws.elastic.cloud Fleet: https://snapshot-c1c61c.es.us-east-1.aws.elastic.cloud Kibana: https://snapshot-c1c61c.kb.us-east-1.aws.elastic.cloud
>>> (linux-amd64-ubuntu-2204-default) Starting SSH; connect with `ssh -i /Users/hinman/src/elastic-agent/.integration-cache/id_rsa [email protected]`
>>> (linux-amd64-ubuntu-2204-default) Connected over SSH
>>> (linux-amd64-ubuntu-2204-default) Preparing instance
>>> (linux-amd64-ubuntu-2204-default) Running apt-get update
>>> Endpoints available: ES: https://snapshot-c1c61c.es.us-east-1.aws.elastic.cloud Fleet: https://snapshot-c1c61c.es.us-east-1.aws.elastic.cloud Kibana: https://snapshot-c1c61c.kb.us-east-1.aws.elastic.cloud
>>> (linux-amd64-ubuntu-2204-default) Install build-essential and unzip
>>> Elasticsearch healthy...
>>> Kibana healthy...
>>> (linux-amd64-ubuntu-2204-default) Failed to install build-essential and unzip; will wait 15 seconds and try again
>>> (linux-amd64-ubuntu-2204-default) Running apt-get update
>>> (linux-amd64-ubuntu-2204-default) Install build-essential and unzip
>>> (linux-amd64-ubuntu-2204-default) Install golang 1.21.8 (amd64)
>>> (linux-amd64-ubuntu-2204-default) Copying repo
>>> (linux-amd64-ubuntu-2204-default) Running make mage and prepareOnRemote
>>> (linux-amd64-ubuntu-2204-default) Copying agent build metricbeat-8.14.0-SNAPSHOT-linux-x86_64.tar.gz
>>> (linux-amd64-ubuntu-2204-default) Waiting for stack to be ready...
>>> (linux-amd64-ubuntu-2204-default) Using Stack with Kibana host https://snapshot-c1c61c.kb.us-east-1.aws.elastic.cloud, credentials available under .integration-cache
>>> (linux-amd64-ubuntu-2204-default) Running sudo tests...
>>> (linux-amd64-ubuntu-2204-default) Test output (sudo) (stderr): go: downloading github.com/rs/zerolog v1.27.0
>>> (linux-amd64-ubuntu-2204-default) Test output (sudo) (stdout): >> go test: remote-linux-amd64-ubuntu-2204-default-sudo.integration Testing
>>> (linux-amd64-ubuntu-2204-default) Test output (sudo) (stdout): exec: gotestsum --no-color -f standard-quiet --junitfile build/TEST-go-remote-linux-amd64-ubuntu-2204-default-sudo.integration.xml --jsonfile build/TEST-go-remote-linux-amd64-ubuntu-2204-default-sudo.integration.out.json -- -tags integration -test.shuffle on -test.timeout 2h -test.run ^(TestBeatsServerless)$ github.com/elastic/elastic-agent/testing/integration
>>> (linux-amd64-ubuntu-2204-default) Test output (sudo) (stderr): go: downloading github.com/gofrs/uuid v4.4.0+incompatible
>>> (linux-amd64-ubuntu-2204-default) Test output (sudo) (stderr): go: downloading github.com/kardianos/service v1.2.1-0.20210728001519-a323c3813bc7
>>> (linux-amd64-ubuntu-2204-default) Test output (sudo) (stderr): go: downloading github.com/jaypipes/ghw v0.12.0
>>> (linux-amd64-ubuntu-2204-default) Test output (sudo) (stderr): go: downloading github.com/schollz/progressbar/v3 v3.13.1
>>> (linux-amd64-ubuntu-2204-default) Test output (sudo) (stderr): go: downloading github.com/gorilla/mux v1.8.0
>>> (linux-amd64-ubuntu-2204-default) Test output (sudo) (stderr): go: downloading github.com/google/go-cmp v0.6.0
>>> (linux-amd64-ubuntu-2204-default) Test output (sudo) (stderr): go: downloading github.com/dolmen-go/contextio v0.0.0-20200217195037-68fc5150bcd5
>>> (linux-amd64-ubuntu-2204-default) Test output (sudo) (stderr): go: downloading go.elastic.co/apm/module/apmgrpc v1.15.0
>>> (linux-amd64-ubuntu-2204-default) Test output (sudo) (stderr): go: downloading golang.org/x/time v0.3.0
>>> (linux-amd64-ubuntu-2204-default) Test output (sudo) (stderr): go: downloading github.com/mitchellh/colorstring v0.0.0-20190213212951-d06e56a500db
>>> (linux-amd64-ubuntu-2204-default) Test output (sudo) (stderr): go: downloading github.com/hectane/go-acl v0.0.0-20190604041725-da78bae5fc95
>>> (linux-amd64-ubuntu-2204-default) Test output (sudo) (stderr): go: downloading github.com/ghodss/yaml v1.0.0
>>> (linux-amd64-ubuntu-2204-default) Test output (sudo) (stderr): go: downloading github.com/jaypipes/pcidb v1.0.0
>>> (linux-amd64-ubuntu-2204-default) Test output (sudo) (stderr): go: downloading github.com/mitchellh/go-homedir v1.1.0
>>> (linux-amd64-ubuntu-2204-default) Test output (sudo) (stdout): -test.shuffle 1710872983145118010
>>> (linux-amd64-ubuntu-2204-default) Test output (sudo) (stdout): ok  	github.com/elastic/elastic-agent/testing/integration	440.526s
>>> (linux-amd64-ubuntu-2204-default) Test output (sudo) (stdout): === Skipped
>>> (linux-amd64-ubuntu-2204-default) Test output (sudo) (stdout): === SKIP: testing/integration TestBeatsServerless/TestSetupPipelines (0.00s)
>>> (linux-amd64-ubuntu-2204-default) Test output (sudo) (stdout): beats_serverless_test.go:276: pipelines only available on filebeat
>>> (linux-amd64-ubuntu-2204-default) Test output (sudo) (stdout): DONE 17 tests, 1 skipped in 519.510s
>>> (linux-amd64-ubuntu-2204-default) Test output (sudo) (stdout): >> go test: remote-linux-amd64-ubuntu-2204-default-sudo.integration Test Passed
>>> Testing completed (17 successful)
>>> Console output written here: build/TEST-go-integration.out
>>> Console JSON output written here: build/TEST-go-integration.out.json
>>> JUnit XML written here: build/TEST-go-integration.xml
>>> Diagnostic output (if present) here: build/diagnostics
--- Clean mage artifacts
>>>> Using ogc instance provisioner
>>>> Using serverless stack provisioner
>>> Destroying serverless stack 8.14.0-SNAPSHOT [stack_id: 8140-SNAPSHOT, deployment_id: c1c61c67869f464e95c718118e150d54]
>>> Bring down instances through ogc

[elastic-sonarqube]

Quality Gate passed

Kudos, no new issues were introduced!

0 New issues
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarQube

[rdner]

Thank you!