[Fleet] Unchecking Enable HTTP endpoint at /liveness does not stop Agent from listening on 6791.

[jerrac]

Kibana version:
8.16.1
Elasticsearch version:
8.16.1
Server OS version:
Ubuntu 22.04
Browser version:
N/A
Browser OS version:
N/A
Original install method (e.g. download page, yum, from source, etc.):
APT
Describe the bug:
Disabling /liveness via agent policy does not stop Agent from listening on localhost:6791
Steps to reproduce:

1. Upgrade to 8.16.1 (Maybe install from scratch as well.)
2. Modify (or create) a policy that does not have Enable HTTP endpoint at /liveness checked.
3. Save the policy and wait long enough for it to be deployed.
4. On the server running agent, run while true; do netstat pln | grep -E '(6791|27910)'; echo '..' ;sleep 5; done;
5. You will see that something is listening on 6791.

FYI, I tried enabling and moving the port to 27910. That worked. It started listening on localhost:27910. But disabling resulted in Agent returning to listening on 6791 like before.

Expected behavior:
Nothing should be listening on 6791.

Screenshots (if relevant):

Any additional context:
For now I can just work around by pointing at an unused port. But still, if the box is unchecked, there should not be anything listening on that port.

• The pull request for the toggles: [Fleet] Add toggles for agent.monitoring.http.enabled and agent.monitoring.http.buffer.enabled to agent policy advanced settings #190984
• Why I was trying to disable in the first place: [Fleet] Add ability to enable and configure HTTP Monitoring #153950 (comment)

BTW, I really appreciate that you added that UI. Being able to change the port is an immense help. Thanks @kpollich for your work!

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[kpollich]

@elastic/elastic-agent - I think we might need some help from the agent team in understanding how to disable listening on this port from this setting.

[fearful-symmetry]

FYI, I tried enabling and moving the port to 27910. That worked. It started listening on localhost:27910. But disabling resulted in Agent returning to listening on 6791 like before.

Interesting. So, it's picking up the config, but not the config to disable it? I'm kind of busy today, but I'll see if I can find time to figure out if this is an agent-side bug or just a config issue between Kibana and agent,.

[fearful-symmetry]

So, I suspect the problem here is that kibana isn't explicitly setting agent.monitoring.http.enabled to false; because agent's default behavior is to enable the monitoring endpoint, it will continue to run unless explicitly disabled. I assume that if we want to actually disable agent from listening on that port, we need to add some checkbox or something that actually sets enabled to false.

[kpollich]

Thanks for looking into this @fearful-symmetry. I think we should explicitly include agent.monitoring.http.enabled: true by default in agent policies generated by Fleet, and then upon unchecking this checkbox swap to explicitly including agent.monitoring.http.enabled: false. Narrowing this down to two possible states instead of true/false/undefined seems like the right thing to do to fix this.

[fearful-symmetry]

@kpollich yeah, agreed, it seems like adding a checkbox or something that defaults to true is the best solution.