Summary

Note: maybe do #423 first

I guess fixes #384, though we can add more alarms. If you have better ideas for what to alarm on, let me know.

Spent a while looking into all the different alarming options:

• ec2 instance alarms; would allow termination of a dead instance (say, CPU redline), but alarm is per-instance, and would have to be recreated when the alarm actually triggered or if the instance is otherwise replaced
• autoscaling scaling alarms; would allow de-scaling and re-scaling, but requires chaining alarms together (say CPU redline -> downscale -> no instances -> upscale) which I didn't really like
• ecs container alarms; what I ended up at

I figured:

• ec2 already does health checks, and I don't really care if the autoscaling group replaces a host; there's nothing we can really do to react to such an alarm
• we've configured health checks on the ecs containers, so they'll be restarted if health checks fail (or if the containers crash)
• autoscaling alarms don't really matter if ecs is already going to restart tasks

So, being notified of unintentionally stopped ecs containers seemed like enough, at least for starters. See tutorial I mostly followed.

Note this doesn't actually add subscriptions, we can do that in follow up commits. It'd be specifying the AlarmSubscription* cloudformation parameters, which would likely be github secrets. We'll need a new webhook to publish to.

Tested manually by creating the topic, rule and subscribing myself. I then ssh'd into the ec2 host and did docker kill. Received a single email for the dead instance. See #duckbutt channel in the beta server.

Checklist

• this is a source code change
  • run format in the repository root
  • run pytest in the repository root
  • link to issue being fixed using a fixes keyword, if such an issue exists
  • update the wiki documentation if necessary
• or, this is not a source code change