Alerts not removed when relation is broken.

[dstathis]

Bug Description

If you remove a relation between a workload and grafana-agent, the alert rules for that workload will remain.

To Reproduce

1. Relate grafana-agent to COS
2. Relate grafana-agent to a workload (eg: zookeeper)
3. Remove the previous relation
4. Check the alerts in prometheus

Environment

                           juju info v0.1                            
┌──────────────┬────────────────────────────────────────────────────┐
│ jhack        │ 0.3.18.3                                           │
│ python       │ 3.10.12 (/home/dylan/repos/jhack/venv/bin/python3) │
│ juju-* snaps │  juju-wait │ 2.8.4~2.8.4 - 96 (stable)             │
│              │  juju      │ 3.1.7-072088c - 25251 (3.1/edge)      │
│ microk8s     │ MicroK8s v1.27.6 revision 6070                     │
│ lxd          │ 5.19                                               │
│ multipass    │ 1.12.2                                             │
│ multipassd   │ 1.12.2                                             │
│ os           │ Ubuntu 22.04.3 LTS                                 │
│ kernel       │ Linux 5.15.0-87-generic x86_64                     │
└──────────────┴────────────────────────────────────────────────────┘

Relevant log output

-

Additional context

No response

[lucabello]

Note: this issue might be caused by Prometheus, so it should be investigated there first.

[sed-i]

Could be because of the way subordinate relation data works: first to the leader and then out to prometheus.