Support using the Elastic Agent integration testing framework to run tests against Serverless projects

[cmacknz]

elastic/elastic-agent#3258 adds the ability to provision Serverless projects in the Elastic Agent integration testing framework, the ability to install standalone Beats on test VMs, and several tests of standalone Beats interacting with Serverless projects.

The scope of this issue is to move these tests to the Beats repository and allow using the Elastic Agent integration testing framework to run them. The Elastic Agent integration testing framework was designed from the beginning to be used from other repositories. This will also enable integration testing specifically between Beats and Elastic Agent whenever the Beats code changes.

To do this we will need to do at least the following tasks:

1. Make the integration testing targets defined in the Elastic Agent magefile importable from Beats.
2. Move the serverless Beats tests to the Beats repository and ensure they can be run manually from the Beats repository.
3. Create a Buildkite pipeline specifically for running these integration tests in the Beats repository.

The agent integration testing framework is currently Elastic licensed and so all builds targets and code related to it in Beats must exist only in the x-pack directory.

[alexsapran]

We should also rethink how the framework deals with serverless projects. The elastic-agent repo, which has a low volume compared to Beats, is hitting the limits; if we were to set these in the Beats, which is a higher traffic repo, we would cause unintended issues.

[cmacknz]

At the time this was written there was a belief that the only way to test serverless was by creating real serverless projects. It appears is actually possible to use the elasticsearch-serverless Docker image to create a local serverless cluster, just with more work than a simple docker run ....

Kibana has done this, see https://github.com/elastic/kibana/blob/14d459f5816ba1531f284a19858967809d853dd8/packages/kbn-es/src/utils/docker.ts#L718-L725 for the implementation.

This could give us a path to running Beats serverless tests against a Docker cluster, which would align much more closely with the way Beats does integration tests today. The primary caveat is the serverless Beats test would have to be rewritten to use this approach.

The Elastic Agent serverless tests might be a bit trickier as they depend on Fleet running beyond just Elasticsearch and Kibana themselves.

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)