Update scrape and remote_write libs for generic HostHealth rules

[MichaelThamm]

Issue

Within the tandem cosl PR, we have a centralized (Prometheus rather than individual charms) way to "inject" alerts on the fly so we can extend Prometheus for generic up/absent rules.

Solution

The prometheus_scrape and prometheus_remote_write libs were updated by injecting generic HostHealth rules to the rules.

Documentation for implementation and testing

Context

In tandem with:

• Enable Generic Alert Rules cos-lib#115
• Centralized rules groups for libraries to import cos-lib#117
• Update cos_agent lib with generic HostHealth rules grafana-agent-operator#232
• Add charmcraft binary-python-packages for Cos-Lib avalanche-k8s-operator#50

Testing Instructions

Without Grafana Agent

1. Deploy Prometheus, Avalanche, and Alertmanager
2. Relate prom:metrics-endpoint avalanche and alertmanager prom:alertmanager
3. Scale Avalanche with juju add-unit avalanche -n 1
4. In the Prometheus UI, check for the new HostHealth rules or query Prometheus with juju show-unit prom/0 | yq -r '."prom/0"."relation-info"[1]."application-data"."alert_rules"' | jq
5. Artificially take down one Avalanche unit with pebble

juju exec --unit avalanche/0 -- \
  PEBBLE_SOCKET=/charm/containers/avalanche/pebble.socket \
  pebble stop avalanche

6. Check the Prometheus UI and notice the HostDown Alert firing (showing the specific unit Avalanche/0)
7. Artificially take down the second Avalanche unit with pebble and check for each Avalanche unit in the labels
8. Check the alerts arriving in Alertmanager for:
   • Host 'prom_8b073ff8-5456-492a-804b-7f9f15c996dc_avalanche_avalanche/0' is down. VALUE = 0 LABELS = map[__name__:up instance:prom_8b073ff8-5456-492a-804b-7f9f15c996dc_avalanche_avalanche/0 job:juju_prom_8b073ff8_avalanche_prometheus_scrape_avalanche-0 juju_application:avalanche juju_charm:avalanche-k8s juju_model:prom juju_model_uuid:8b073ff8-5456-492a-804b-7f9f15c996dc juju_unit:avalanche/0]

With Grafana Agent

Scrape

1. Deploy Grafana Agent, Prometheus, Avalanche, and Alertmanager
2. Relate gagent:metrics-endpoint avalanche, gagent:send-remote-write prom, and alertmanager prom:alertmanager
3. Scale Avalanche with juju add-unit avalanche -n 1
4. In the Prometheus UI, check for the new HostHealth and AggregatorHostHealth rules or query Grafana Agent with juju show-unit gagent/0 | yq -r '."gagent/0"."relation-info"[0]."application-data"."alert_rules"' | jq
5. Artificially take down one Avalanche unit with pebble
6. Check the Prometheus UI and notice the HostDown Alert firing (showing the specific unit Avalanche/0)
7. Artificially take down the second Avalanche unit with pebble and check for each Avalanche unit in the labels
8. Check the alerts arriving in Alertmanager (similar result as without Grafana Agent).

Remote write

1. Artificially take down Grafana Agent with pebble

juju exec --unit gagent/0 -- \
  PEBBLE_SOCKET=/charm/containers/agent/pebble.socket \
  pebble stop agent

2. Check the Prometheus UI and notice the HostUnavailable Alert firing in both the HostHealth and AggregatorHostHealth groups

Note

This does not show each unit, with 2 avalanche units, alert labels shows once per app:
alertname=HostUnavailablejuju_application=avalanchejuju_charm=avalanche-k8sjuju_model=promjuju_model_uuid=8b073ff8-5456-492a-804b-7f9f15c996dcseverity=critical

3. Check the alerts arriving in Alertmanager for:
   • Metrics not received from host ''. VALUE = 1 LABELS = map[juju_application:avalanche juju_model:prom juju_model_uuid:8b073ff8-5456-492a-804b-7f9f15c996dc]

Cos-proxy

1. Copy lib/charms/prometheus_k8s/v0/prometheus_scrape.py into cos-proxy and pack the charm.
2. Deploy the bundles:

k8s (in a model named "prom")

bundle: kubernetes
applications:
  alertmanager:
    charm: alertmanager-k8s
    channel: latest/stable
    revision: 138
    scale: 1
    trust: true
  prom:
    charm: prometheus-k8s
    channel: latest/stable
    revision: 221
    scale: 1
    trust: true

--- # overlay.yaml
applications:
  prom:
    offers:
      prom:
        endpoints:
        - metrics-endpoint
        - receive-remote-write
        acl:
          admin: admin

lxd

series: jammy
saas:
  prom:
    url: microk8s:admin/prom.prom
applications:
  cp:
    # charm path is relative to the bundle file
    charm: ./cos-proxy_ubuntu-22.04-amd64.charm
    series: jammy
    scale: 1
  nrpe:
    charm: nrpe
    channel: edge
  ub:
    charm: ubuntu
    channel: edge
    scale: 1
relations:
- - nrpe:monitors
  - cp:monitors
- - ub:juju-info
  - nrpe:general-info
- - cp:downstream-prometheus-scrape
  - prom:metrics-endpoint

3. Stop the vector service in cos-proxy with juju exec --unit cp/0 "sudo systemctl stop vector"
4. Check the Prometheus UI for the alert:
   

Upgrade Notes

By fetching the new libs you would get a set of new alerts automatically. If charms already had up/absent alerts, this will result in duplication of alerts and rules.

• up/absent alerts are ubiquitous and are handled by the libs modified in this PR. Any custom alerts duplicating this behaviour can be removed.

With the new design introduced in this PR, you would get a separate HostUnavailable alert for Grafana Agent itself and each unit that is aggregated by it.