Rework runtime manager updates to block the coordinator less

[faec]

What does this PR do?

Fix #3738, which happens when Coordinator and runtime.Manager block on a policy update and state reporting gets backed up.

Summary of changes:

• Give the runtime manager an explicit run loop in its own function instead of managing all changes via mutex
• Manager.Update now sends the component model update to the runtime manager loop via a channel
• Component model updates are triggered asynchronously from the runtime manager loop, so neither Coordinator nor the runtime manager block their communication while an update is in progress
• Results from the update process are sent back to Coordinator on the runtime manager's (previously unused) error channel instead of returning directly from Update
  • on the Coordinator side, the internal variables for the runtime manager error and the update error have been merged
• The runtime manager now uses a done channel to signal that it is shutting down and will no longer accept incoming requests
• (adjacent cleanup) some unneeded / no longer needed synchronization handling has been removed from the runtime manager (updateMx which is no longer needed since (*Manager).runLoop ensures update calls do not conflict, netMx which was only used to get the tcp listener port, and some startup / shutdown atomics)

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

Related issues

• Fixes runtime.update can block all checkins and state processing while waiting for shutdown #3738
• Fixes Linux agents gets unhealthy on enabling/disabling modules for System/Linux integration. #3654
• Fixes Agent gets unhealthy on assigning from policy with Elastic Defend integration to without Defend integration. #3617
• Fixes https://github.com/elastic/ingest-dev/issues/2560
• Fixes Agent with Elastic Defend remains unhealthy for long when invalid configuration is added for sometime and then updated with correct. #3721
• Fixes Agent gets unhealthy temporarily for a long time on removing and re-adding Elastic Defend integration. #3507

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[mergify]

This pull request does not have a backport label. Could you fix it @faec? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v./d./d./d is the label to automatically backport to the 8./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-11-16T17:31:34.988+0000

• Duration: 14 min 17 sec

❕ Flaky test report

No test was executed to be analysed.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages.

• run integration tests : Run the Elastic Agent Integration tests.

• run end-to-end tests : Generate the packages and run the E2E Tests.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[pchila]

Code looks good, I think that the suggestion I left should take carel of the deadlock we have in unit test ;)

[blakerouse]

Looks really good.

I like the removal of the locks and the fact that the error now comes through the errCh. Unit testing has good coverage with the change as well.

[faec]

There's still an issue with this PR -- in manual testing it gets stuck forever during the install command, enrollment reaches Kibana but is never properly reported Agent-side. That might also be why the integration tests seem stalled without a result on CI. Investigating.

[faec]

Found the problem: Coordinator blocks to tell the runtime manager about the new update, and the runtime manager blocks to tell Coordinator about the result of the update. Usually these two aren't in conflict, but during an install two Update calls happen right after each other, so both objects end up waiting on each other.

I still ran the update logic synchronously on the runtime manager goroutine with the idea that it was a "less drastic change" that could be done in a smaller first-pass PR, but I think I just need to fix the blocking entirely.

[faec]

New version is ready that fixes the full issue instead of just mitigating it. I had to add some more state handling variables and code but I've tried to comment thoroughly, and this should prevent any number of policy changes from ever blocking state update reporting.

[ycombinator]

LGTM. Very easy to follow the new implementation and thanks for the adjacent cleanup as well.

[pchila]

LGTM

[elastic-sonarqube]

SonarQube Quality Gate

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

85.5% Coverage
0.0% Duplication