Metrics not collected for Config-Reloader sidecars to Alertmanager and Prometheus

[gsmith-sas]

After enabling TLS by default as part of our 1.2.15 (18JUL23) release, metrics are no longer being collected from the config-reloader sidecar containers associated with the Alertmanager and Prometheus pods deployed as part of the metric monitoring components of this project. A recently released update to the kube-prometheus-stack Helm chart will allow us to address this issue in a coming update. This is a minor issue and most users will not be aware of it unless they review the Targets screen in the Prometheus web app. These two targets will be reported as "unhealthy" on that screen.

The following instructions describe how to manually patch the service monitor definitions to allow Prometheus to collect metrics from these two components.

• Edit the definition for the v4m-prometheus serviceMonitor resource (deployed by default in the monitoring namespace). This can be done using the kubectl edit command (e.g. kubectl -n monitoring edit servicemonitor v4m-prometheus) or through an interactive Kubernetes management tool such as Lens.

• Update the stanza defining the metric endpoint for the config-reloader sidecar. Note that the serviceMonitor defines two metric endpoints: one for the primary container and the other for the config-reloader sidecar. The config-reloader sidecar metric endpoint is the generally the second one and can be identified because it refers to the reloader-web port. The updated definition of this endpoint is shown below:

- path: /metrics
  port: reloader-web
  scheme: https
  tlsConfig:
    insecureSkipVerify: true

• The changes to make are
  • change the scheme from "http" to "https"
  • add the tlsConfig
  • add the insecureSkipVerify key and set its value to "true"
  • Remember that YAML is very sensitive to indentation and ensure the keys are indented properly.
• After saving the changes, it may take a few (2-3) minutes before they are picked up and Prometheus.
• Repeat the same steps with the definition of the service monitor for v4m-alertmanager.

NOTE: These steps were validated with test clusters running the Kube-Prometheus Stack version 56.6.2 which is the version deployed as part of our most recent version 1.2.23 released on 19MAR24.

[gsmith-sas]

This problem was fixed by PR #656 and the updated code is now available as part of our 1.2.27 release. Closing issue.