Cloudwatch Agent metrics endpoint unavailable

[krisko]

Describe the bug
When running amazon-cloudwatch-observability agent in EKS (installed as AWS addon) the agent has prometheus scrape annotations and creates cloudwatch-agent-monitoring service. This service should return prometheus metrics exposed by the agent but instead the call just fails to connect

$ curl cloudwatch-agent-monitoring.amazon-cloudwatch.svc.cluster.local:8888/metrics

curl: (7) Failed to connect to cloudwatch-agent-monitoring.amazon-cloudwatch.svc.cluster.local port 8888 after 4 ms: Could not connect to server

Annotations on the agent pods:

│                   prometheus.io/path: /metrics
│                   prometheus.io/port: 8888
│                   prometheus.io/scrape: true

Steps to reproduce
Install amazon-cloudwatch-observability addon (addon_version that has been tested "v1.7.0-eksbuild.1" and "v1.10.0-eksbuild.2")

What did you expect to see?
After querying the /metrics endpoint agent should return prometheus compatible metrics. Example below shows sample output from different metrics endpoint:

$ curl opentelemetry-operator.opentelemetry-operator-system.svc.cluster.local:8080/metrics
# HELP certwatcher_read_certificate_errors_total Total number of certificate read errors
# TYPE certwatcher_read_certificate_errors_total counter
certwatcher_read_certificate_errors_total 0
# HELP certwatcher_read_certificate_total Total number of certificate reads
# TYPE certwatcher_read_certificate_total counter
certwatcher_read_certificate_total 1

What did you see instead?
Error message [Could not connect to server](curl: (7) Failed to connect to cloudwatch-agent-monitoring.amazon-cloudwatch.svc.cluster.local port 8888 after 4 ms: Could not connect to server)

What version did you use?
Version: "v1.7.0-eksbuild.1" and "v1.10.0-eksbuild.2"

What config did you use?
Config: default configuration, without any additional values

Environment
OS: EKS cluster v1.29

[jefchien]

Related to aws/amazon-cloudwatch-agent-operator#190

[github-actions]

This issue was marked stale due to lack of activity.

[github-actions]

Closing this because it has stalled. Feel free to reopen if this issue is still relevant, or to ping the collaborator who labeled it stalled if you have any questions.