Create Prometheus Exporter

[TobyRet]

Add Prometheus exporter to the application.

This will allow us to create and capture application metrics which can then:

• become searchable via Prometheus Querying.
• generate alerts
• graphed via Granfana

[jennyd]

@TobyRet and I just had a chat about this - main points to think about are:

• running an exporter per replica rather than one for all of them, so we can get separate application metrics for each replica
• we'd prefer to not have our metrics endpoint visible to everyone on the internet, but Prometheus still needs to be able to reach it - does that mean setting up auth on it?
• docs/script for running the exporter in dev, so that we can easily see what our metrics are doing and work on adding new ones with confidence

[alkar]

To be clear, I've recommended running this as a stand-alone Deployment only if the metrics are going to always be the same in each replica, thinking it might be easier to manage. It does not have to be done that way and it would be fine to export duplicate metrics from a number of replicas.

As @jennyd points out, if you want to export additional application metrics (eg. response times) then it's obviously preferable to export from the actual application, from each replica.

To prevent metrics from leaking to the internet, the most common design I've seen is to start another http listener on a separate port that is not exposed via the Ingress. Alternatively, if the app structure allows it, you can specify paths in the Ingress rules (eg. expose /api only ).

[jennyd]

Also, it's definitely worth keeping an eye on prometheus/client_ruby#95 - if the official ruby client starts supporting pre-fork servers then we should consider switching over to use that 🎉

[TobyRet]

There is now one Prometheus client pre replica as discussed