Allow installing a second agent on the same machine for development

[cmacknz]

• Closes https://github.com/elastic/ingest-dev/issues/3260
• Depends Wait on the watcher at startup instead of just releasing resources associated with it #4834

Allows installing or running a second Elastic Agent on the same machine for development using the --develop argument. The core problem this PR aims to solve is allowing agent developers to install a second agent on their machines when there is also an InfoSec managed agent installed. Once installed, the second agent is available with the elastic-development-agent command. I put development in the middle instead of at the end because it makes tab based autocompletion in the shell easier.

I have manually tested that the development agent can enroll with Fleet and upgrade. The changes needed for a second agent are isolated to install time and interacting with the daemon. Features like upgrades and enrolling in Fleet require no changes because they are contained into the agent's installation directory. For this reason I have only added integration tests to ensure installing a second agent alongside an existing one works. We shouldn't need to test anything else as there are no features that reach outside the agent installation directory.

I sanity checked what happens if you install Defend a second time on the same machine to make sure nothing explodes, and it fails immediately in a nice place before the second install is even attempted:

sudo elastic-development-agent status
┌─ fleet
│  └─ status: (HEALTHY) Connected
└─ elastic-agent
├─ status: (DEGRADED) 1 or more components/units in a failed state
└─ endpoint-default
    ├─ status: (FAILED) failed to start connection info service endpoint: failed to start connection credentials listener: listen tcp 127.0.0.1:6788: bind: address already in use
    ├─ endpoint-default
    │  └─ status: (FAILED) failed to start connection info service endpoint: failed to start connection credentials listener: listen tcp 127.0.0.1:6788: bind: address already in use
    └─ endpoint-default-53f5058f-8a9f-460d-b36c-5d4db6131658
        └─ status: (FAILED) failed to start connection info service endpoint: failed to start connection credentials listener: listen tcp 127.0.0.1:6788: bind: address already in use

Reproducing the added README section for instructions on how to install and test:

Development Installations

⚠️ Development installations are not officially supported and are intended for Elastic Agent developers.

If you are an Elastic employee, you already have an Information Security managed Elastic Agent installed on your machine for endpoint protection. This prevents you from installing the Elastic Agent a second time for development without using a VM or Docker container. To eliminate this point of friction, Elastic Agent has a development mode that permits installing the Elastic Agent on your machine a second time:

# All other arguments to the install command are still supported when --develop is specified.
sudo ./elastic-agent install --develop
# The run command also supports the --develop option to allow running without installing when there is another agent on the machine.
./elastic-agent run -e --develop

Using the --develop option will install the agent in an isolated Agent-Development agent directory in the chosen base path.

Development agents enrolled in Fleet will have the Development tag added automatically. Using the default base path on MacOS you will see:

sudo ls /Library/Elastic/
Agent
Agent-Development

The elastic-agent command in the shell is replaced with elastic-development-agent to interact with the development agent:

# For a privileged agent
sudo elastic-development-agent status
# For an unprivileged agent
sudo -u elastic-agent-user elastic-development-agent status

The primary restriction of --develop installations is that you cannot run Elastic Defend a second time on the same machine. Attempting toinstall Defend twice will fail with resource conflicts. All other integrations should be usable provided conflicting configurations are changed ahead of time. For example two agents cannot bind to the same agent.monitoring.http.port to expose their monitoring servers

[mergify]

This pull request does not have a backport label. Could you fix it @cmacknz? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v./d./d./d is the label to automatically backport to the 8./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b second-agent-same-machine upstream/second-agent-same-machine
git merge upstream/main
git push upstream second-agent-same-machine

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b second-agent-same-machine upstream/second-agent-same-machine
git merge upstream/main
git push upstream second-agent-same-machine

[elastic-sonarqube]

Quality Gate failed

Failed conditions
31.3% Coverage on New Code (required ≥ 40%)

See analysis details on SonarQube

[alexsapran]

that would be great, as this is something that would be helpful with the work I am doing.

Having the ability to just download an elastic-agent and just run it locally without installing it.

[blakerouse]

Looks good thanks for the fixes and the exposing of --namespace optional as a command line option.

[cmacknz]

31.3% Coverage on New Code (required ≥ 40%)

I am going to force merge past this, most of the uncovered code is from the conversion of variables into functions and the uncovered code is tested in integration tests.