Elastic Agent always starts http monitoring service even if metrics are disabled

[JanKnipp]

Blocked by: elastic/kibana#168629

Were facing an issue when the port (6791) being used by elastic agent for metrics is already in use in a fleet regulated agent scenario. We a running a backup solution which was not able to run due to the use of the same port.

As configuration of the ports (6791/http & 6789/grcp) does not seems to be possible for a fleet controlled agent our idea was do disable
"Agent Monitoring" in the agent policy.

After disabling it in the policy "elastic-agent inspect" reports that
monitoring, logs & metrics
are disabled.

My expectation would be that this would also trigger the http endpoint to be disabled as it is a subset of agent.monitoring

At least thats what im interpreting based on the elastic-agent-reference sample.

# agent.monitoring:
# # enabled turns on monitoring of running processes
# enabled: false
# # enables log monitoring
# logs: false
# # enables metrics monitoring
# metrics: false
# # exposes /debug/pprof/ endpoints
# # recommended that these endpoints are only enabled if the monitoring endpoint is set to localhost
# pprof.enabled: false
# # exposes agent metrics using http, by default sockets and named pipes are used
# http:
# # enables http endpoint
# enabled: false
# # The HTTP endpoint will bind to this hostname, IP address, unix socket or named pipe.
# # When using IP addresses, it is recommended to only use localhost.
# host: localhost
# # Port on which the HTTP endpoint will bind. Default is 0 meaning feature is disabled.
# port: 6791
# # Metrics buffer endpoint
# buffer.enabled: false

In the end netstat shows that both ports for http and grpc are still open which prevents us from using the agent at all.

• Version: Elastic Agent 8.7.1
• Operating System: Windows (Windows Server 2019), Linux (RHEL7)

[cmacknz]

The gRPC port can't be disabled as it is used for internal state changes and health checks. The HTTP port should be disabled here, that seems like a bug.

[it4ewa]

Dear Elastic team,

are there any updates on that? This is currently a show-stopper for the roll-out of the Agent.

thanks and best regards

[pierrehilbert]

Hello @it4ewa,
Could you give me more details about why it's a show-stopper?
I understand that it's not convenient to have the HTTP port enabled but I didn't put a high priority on this because it's not breaking anything from my perspective.
Thanks

[it4ewa]

Hi @pierrehilbert ,
the problem is, that by the moment we deploy the agent on hosts where also the back up solution is running, the back up solution stops working.

[hamidallaoui]

Hi @it4ewa,

Did you find a solution because we are facing the same issue with conflict on port 6791 used by another application.

We would like Elastic Agent to not use that port and replace it for example by 6790.

Thanks,

[it4ewa]

Hi @it4ewa,

Did you find a solution because we are facing the same issue with conflict on port 6791 used by another application.

We would like Elastic Agent to not use that port and replace it for example by 6790.

Thanks,

Hi @hamidallaoui ,

here is the workaround I got from elastic support:

Enroll the Elastic Agent as usual, it will configure the monitoring port, which is problematic
On the Agent machine, there is a elastic-agent.yml. This file won’t be taken into account after an Agent is enrolled via Fleet
However, any changes in that file will be applied to the Agent upon re-enrollment
As such, after enrolling the Agent, you can edit the elastic-agent.yml locally, change the monitoring port and then re-enroll the Agent into Fleet
First, unenroll the Agent (can only be done from the Fleet side): https://www.elastic.co/guide/en/fleet/current/unenroll-elastic-agent.html
Update the local elastic-agent.yml file with the desired monitoring port
Then, re-enroll the Agent to Fleet on the client side. Either via the enroll or the install command as per https://www.elastic.co/guide/en/fleet/current/elastic-agent-cmd-options.html#elastic-agent-enroll-command

@ Elastic Team:

Do you have any news about when this bug will be fixed (version or date)?

[michalpristas]

reopening issue, it seems that some use cases are counting on this being present. we will figure out alternative solution

[cmacknz]

So looking at the description of the issue in #2734 along with #2509 as it is related we need the following to be true:

1. It is possible to disable the HTTP monitoring server. Right now it is always enabled.
2. The HTTP monitoring server should only be started if agent.monitoring.enabled: true and agent.monitoring.http.enabled: true.
3. The HTTP monitoring server should bind to localhost by default when enabled as described in Agent monitoring server is binding to all interfaces instead of localhost #2509. This is the intent looking at the agent.monitoring.http.host value

   elastic-agent/elastic-agent.reference.yml

   Line 154 in 5071cfa

   # host: localhost

4. It is possible to configure the host and port the HTTP monitoring server binds to. This should already work per the agent.monitoring.http configuration above.

Making the above true should result in this functionality behaving in a consistent and reasonable way, and fixes the security problem of the agent monitoring server potentially being exposed to a public interface by resolving #2509.

We have two additional problems with this implementation:

1. The HTTP monitoring server is used in Elastic Cloud, which currently relies on it being enabled by default. We will need to change the configuration used in cloud to set agent.monitoring.enabled: true, agent.monitoring.http.enabled: true, and likely set agent.monitoring.http.host: "0.0.0.0" to bind to all interfaces as it did before prior to making this change. This should be possible and work exactly the same as it does today, except it will do so explicitly.
2. None of these parameters are exposed in Fleet today. However this no longer matters because the Fleet policy override API now exists to allow users to change this even if it isn't in the UI. [Fleet] Add agent policy API to add settings not yet supported by UI kibana#158699 (comment)

So I think we should address this along with #2509 at the same time because they will both require a configuration change in cloud as a prerequisite.

Ideally we would also expose all of the monitoring related parameters in Fleet but doing that doesn't block this implementation, because the override API allows changing these parts of the policy regardless.

[cmacknz]

After some more investigation by Michal, we need to change the pre-configured policy in Kibana so that this won't be a breaking change in cloud. This is now blocked by elastic/kibana#168629.

[hamidallaoui]

Hi @it4ewa,
Did you find a solution because we are facing the same issue with conflict on port 6791 used by another application.
We would like Elastic Agent to not use that port and replace it for example by 6790.
Thanks,

Hi @hamidallaoui ,

here is the workaround I got from elastic support:

Enroll the Elastic Agent as usual, it will configure the monitoring port, which is problematic On the Agent machine, there is a elastic-agent.yml. This file won’t be taken into account after an Agent is enrolled via Fleet However, any changes in that file will be applied to the Agent upon re-enrollment As such, after enrolling the Agent, you can edit the elastic-agent.yml locally, change the monitoring port and then re-enroll the Agent into Fleet First, unenroll the Agent (can only be done from the Fleet side): https://www.elastic.co/guide/en/fleet/current/unenroll-elastic-agent.html Update the local elastic-agent.yml file with the desired monitoring port Then, re-enroll the Agent to Fleet on the client side. Either via the enroll or the install command as per https://www.elastic.co/guide/en/fleet/current/elastic-agent-cmd-options.html#elastic-agent-enroll-command

@ Elastic Team:

Do you have any news about when this bug will be fixed (version or date)?

Hi @it4ewa,

Thanks for the given workaround, actually we already applied that after an advice from a colleague and it is working fine.

But the better solution will be to be able to change any listening port used by elastic agent without having to re-enroll it with the fleet.

Thanks & Best Regards,

Hamid Allaoui