Include documentation on how to run the alertmanager with Prometheus

[hemajv]

I have the Github alertmanager receiver running locally, and I have a prometheus instance deployed. How do I route my alerts from prometheus to my locally running github receiver? I couldn't find a good documentation on how I can do this.

Additional documentation on the various arguments supported for the github receiver along with examples on how to configure them would be useful!

(This is related to our work in operate-first/sre#19)

[stephen-soltesz]

Hello, @hemajv I can provide some hints based on our use-case.

The basic flow of alerts is: prometheus (generates alerts) -> alertmanager (routes alerts) -> github-receiver (receives alert)

This is our deployment of the github-receiver in k8s: https://github.com/m-lab/prometheus-support/blob/master/k8s/prometheus-federation/deployments/github-receiver.yml

The main setup is to create a personal github auth token with permission to create issues in your target repo(s). Otherwise the reciever just waits to get alert events from the alertmanager.

The k8s config also includes a service definition: https://github.com/m-lab/prometheus-support/blob/master/k8s/prometheus-federation/services/github-receiver-service.yml

which produces a name usable in-cluster like:

GITHUB_RECEIVER_URL=http://github-receiver-service.default.svc.cluster.local:9393/v1/receiver

That URL should be added to the alertmanager config, such as in:
https://github.com/m-lab/prometheus-support/blob/master/config/federation/alertmanager/config.yml.template#L139

We have similar configs to point prometheus at the alertmanager: https://github.com/m-lab/prometheus-support/blob/master/config/federation/prometheus/prometheus.yml.template#L30

Please let us know if you have additional questions or suggestions for what kind of documentation would be most helpful. I imagine that everyone's deployment environment will be a little different. But there are a lot of pieces, so I understand the need to have better breadcrumbs.

[hemajv]

@stephen-soltesz thank you so much for providing the additional documentation/guides! We were able to deploy and test our Prometheus alerts with the GitHub alertmanager receiver 😃

Another question I had was, how do I modify the alert message in the issue created in the repo? For eg, in this issue, how do I modify the URLs being displayed and also is it possible to tag someone to the issue?

[stephen-soltesz]

Hello, @hemajv -- Great, I'm glad to hear you've got things running!

Right now the issue template is static, and defined here:
https://github.com/m-lab/alertmanager-github-receiver/blob/master/alerts/template.go#L54

We would be happy to review/accept changes that make the template an input parameter while preserving default behavior.

For automated assignment, I recommend searching for github automation solutions. For example, https://github.com/apps/auto-assign-issues or https://github.com/marketplace/actions/auto-assign-issue (note the very subtle name difference). I've not used these personally, and there are many other options than just these. These are just examples of what is a common use-case that I'm sure you can find to meet your needs better than a feature in the github-receiver could.

[hemajv]

Hi @stephen-soltesz, another question I had was how do we remove the in progress label to an alert when it has been marked as resolved? For eg: https://github.com/operate-first/SRE/issues/119, this issue has been resolved and closed but it still contains the in progress label.

[hemajv]

And the links in the issues are not clickable, is there a way we can modify/define the appropriate URLs in the issues?
(see: https://github.com/operate-first/SRE/issues/119#issuecomment-795477922)

[stephen-soltesz]

@hemajv additional labels may be specified as flags in the github-reciever configuration -- see: https://github.com/operate-first/apps/blob/master/odh/base/monitoring/github-receiver.yaml#L32 This adds the label automatically when the issue is created, but they are never removed by the github-receiver. The idea was that these labels would be part of an interface to down stream automation (like the label automation links above), or simply handled manually by a person.

The links in https://github.com/operate-first/SRE/issues/119 look like they're k8s service addresses. These allow services to talk to one another within the k8s cluster, but as you've found do not work outside the cluster. These are the service names configured in prometheus and known to the alertmanager.

See for example the --web.external-url=WEB.EXTERNAL-URL flag for alertmanager and prometheus.

docker run -it prom/alertmanager --help
docker run -it prom/prometheus --help

[hemajv]

@stephen-soltesz I see, so I want to update the alertmanager URL in the issue to point to our alertmanager and would like to update firing to point to the active prometheus alert URL: http://prometheus-portal-opf-monitoring.apps.zero.massopen.cloud/alerts. So is there a way we can update our issue template directly or do we provide additional flags while running the GitHub receiver like the one you mentioned --web.external-url=WEB.EXTERNAL-URL?

[stephen-soltesz]

Using the existing version of the github-reciever, it would be necessary to use the --web.external-url flags to make sure clickable links are plumbed through.

I would be open to PRs that make the currently static issue template configurable, e.g. read from a file. https://github.com/m-lab/alertmanager-github-receiver/blob/master/alerts/template.go#L54 However, I won't be able to do this myself anytime soon. But, reviews are no problem.

[hemajv]

@stephen-soltesz thank you for the clarification! So as I understand it, we will need to add the --web.external-url flag to our alertmanager.yaml and to prometheus.yaml, will try this out 👍