Change upgrade watcher to use StateWatch

[blakerouse]

What does this PR do?

Refactors how the upgrade watcher works removing the need to parse the PID from any system service manager.

The watcher now tracks the inability to connect to the daemon, lost connections to the daemon, agent failures, component failures, and flip-flopping of healthy/error states. The tracking of the lost connections to the daemon allows the removal of the PID tracking that used to be present. The error tracking is more robust as it requires that an error be an error for at least 30 seconds, allowing the Elastic Agent to possibly have an issue and then recover (before if the State call happened when an error was present it would just become failed).

This also includes unit testing for the AgentWatcher service that previously had no unit tests.

Why is it important?

Completely removes the requirement of executing and reading output of service managers (removing the need for root access). Improves the overall logic and provides better testing.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• [ ] I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test (covered by existing tests)

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-10-18T20:41:10.213+0000

• Duration: 25 min 25 sec

Test stats 🧪

Test	Results
Failed	0
Passed	6513
Skipped	59
Total	6572

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages.

• run integration tests : Run the Elastic Agent Integration tests.

• run end-to-end tests : Generate the packages and run the E2E Tests.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[elasticmachine]

🌐 Coverage report

Name	Metrics % (covered/total)	Diff
Packages	98.81% (83/84)	👍
Files	66.997% (203/303)	👎 -0.109
Classes	65.946% (366/555)	👍 0.227
Methods	53.128% (1155/2174)	👍 0.131
Lines	39.542% (13633/34477)	👍 0.28
Conditionals	100.0% (0/0)	💚

[ycombinator]

elastic-agent/testing/upgradetest/watcher.go

Line 22 in defde80

crash_check.interval: 15s

Should we be removing this line? Or do we need to leave it in there for older Agent versions (that don't have the watcher implementation in this PR)?

[ycombinator]

It looks like we only increment failedCount when we received a non-nil error from failedCh but previously there was no error (i.e. failedErr == nil). So that means failedCount actually represents the number of times we flip-flopped from a non-error state to an error state, right? If my understanding is correct, could we rename this to something like flipFlopCount or something else indicating that it's counting flip-flops?

[blakerouse]

Done.

[blakerouse]

elastic-agent/testing/upgradetest/watcher.go

Line 22 in defde80

crash_check.interval: 15s

Should we be removing this line? Or do we need to leave it in there for older Agent versions (that don't have the watcher implementation in this PR)?

This has to remaining for older versions.

[ycombinator]

LGTM.

[blakerouse]

buildkite test this

[blakerouse]

another unrelated failure... this time the endpoint integration

[blakerouse]

buildkite test this

[elastic-sonarqube]

SonarQube Quality Gate

0 Bugs
0 Vulnerabilities
0 Security Hotspots
2 Code Smells

67.4% Coverage
6.3% Duplication