Extract creating of ESS deployment for integration tests in a separate mage target

[rdner]

Describe the enhancement:
There should be a separate mage target for creation of the ESS deployment that we use for integration testing.

Describe a specific use case for the enhancement or feature:

We should be able to do it separately to running the actual tests, so we are more flexible with how we run the tests.

What is the definition of done?

There is a mage target that runs an ESS deployment according to the integration test requirements.

• Refactor StatefulProvisioner
• Refactor ServerlessProvisioner
• Refactor InstanceProvisioner
• Create unit tests for refactored provisioners

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[pazone]

I'm trying to extract an independent createEssStack mage target that will just create a stack and write its state for later usage.
I stumbled upon a problem that the states (OGC instances and stacks) are managed by the runner.go. So testing/runner package does too many things:

• manages OGC and stack instances states
• Contains interfaces like StackProvisioner and Stack
• runs tests etc.

The only solution I found is to delegate stack provisioning to the ess package. The runner.go will only have a link to StackProvisioner instance that will manage stacks and their states.
To achieve that I would move the StackProvisioner interface and all stack-related logic to the ess package. For consistency, the OGC instance provisioning and state logic would be moved to the ogc package.
That's a serious change. From my perspective, this change would make our codebase more straightforward it will reduce the responsibility scope for ess, ogc and testing/runner packages.

@blakerouse @cmacknz @rdner I would appreciate your ideas. Do you happen to know an easier way to achieve that?

[blakerouse]

The only solution I found is to delegate stack provisioning to the ess package. The runner.go will only have a link to StackProvisioner instance that will manage stacks and their states. To achieve that I would move the StackProvisioner interface and all stack-related logic to the ess package. For consistency, the OGC instance provisioning and state logic would be moved to the ogc package. That's a serious change. From my perspective, this change would make our codebase more straightforward it will reduce the responsibility scope for ess, ogc and testing/runner packages.

Do not move StackProvisioner to the ess package, it is not ess specific. In the future other stack provisioners might be added, like a local elastic-package provisioner that uses the elastic-package CLI to create a stack. You can move it to a common package if you like seperate from the runner. If you do that best to move other common interfaces to that package as well.

I don't see any OGC specific things in runner package except for some old comments that are not accurate. Looks like those should be updated and where missed when support for multiple runners where added.

Another option would be to add a noop provisioner for the InstanceProvisioner then pass that into the runner, where it will do nothing for creating instances, but still create the stack.

[pazone]

Thanks for the advice!

I don't see any OGC specific things in runner package except for some old comments that are not accurate. Looks like those should be updated and where missed when support for multiple runners where added.

I mostly mean the State. It keeps both: instances and stacks states. I think it makes sense to decouple the states to make the ESS stack provisioner independent

[pazone]

As we agreed on the Eng Prod Office hours I migrated the ESS provisioning logic to terraform. It now looks much simpler. To test it I'm running the integration tests directly on BK agents. It's a bit out of scope but it's easier to test the terraform this way. For not the ESS stack is created and destroyed as planned. However, the tests are failing due to batching misconfiguration. Now I'm trying to run integration tests with a simple go test command

[pazone]

Integration tests are running on BK agents. I have a problem with sudo tests. The agent hung up when I switched the user to the root. Solving that

[pazone]

Had to create Basic custom VM images for elastic-agent to add buildkite-agent to proper groups. https://github.com/elastic/ci-agent-images/pull/798

[pazone]

The ESS deployment creation is extracted to a different script file. And it's proven that the terraform managed deployment works for integration tests. The deployment is being successfully deleted after the tests execution. We can wrap the script into a mage target or use some golang terraform API.
It's done in a large PoC Pull request and can be integrated to the testing flow when it's merged