Use Shutdown to gracefully halt HTTP servers

[michel-laterman]

What is the problem this PR solves?

Checkin API long-poll connections terminate with an error when the server needs to shut down (providing a 5xx to the client).

How does this PR solve the problem?

Use the Shutdown method with a timeout to try to gracefully shutdown
HTTP connections. If the server times out while attempting to drain the
connections, connections are forced closed. If there is a long-polling
checkin request, the server should send a 200 response with the ack
token used in the request (same behaviour as the poll completing
normally).

Design Checklist

• I have ensured my design is stateless and will work when multiple fleet-server instances are behind a load balancer.
• I have or intend to scale test my changes, ensuring it will work reliably with 100K+ agents connected.
• I have included fail safe mechanisms to limit the load on fleet-server: rate limiting, circuit breakers, caching, load shedding, etc.

Checklist

• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool

Related issues

• Closes Gracefully drain checkin requests on shutdown #2902

[michel-laterman]

Does anyone know why the linter is all of a sudden having issues with our apm imports?

[jlind23]

@michel-laterman Looks like the linter is ok and only Sonar analysis complains about the insufficient code coverage.
Here is the coverage report - Profile and handlecheckin doensn't have any test. I let you decide whether or not this is a real problem.

[michel-laterman]

The changes to handleCheckin should be covered by the new integration test (but coverage for those tests don't work as expected). I've also added a test case for the profiler

[jsoriano]

Code-wise LGTM, not tested.

[jsoriano]

Nit. Could it be closed here?

Suggested change

	resp, err = http.DefaultClient.Do(req) //nolint:bodyclose // closed outside the loop
	if err == nil {
	resp, err = http.DefaultClient.Do(req)
	if err == nil {
	resp.Body.Close()

[jlind23]

buildkite test it

[jlind23]

@nchaulet @michel-laterman all tests passed, shall we merge this?

[nchaulet]

Tested locally and it seems to work as expected 🚀

[elastic-sonarqube]

Quality Gate passed

The SonarQube Quality Gate passed, but some issues were introduced.

1 New issue
0 Security Hotspots
57.1% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube