Enable Prometheus metrics exporter for ingress-nginx controller.

[achton]

What does this PR do?

To better track how the ingress-nginx controller is doing, we should export metrics from it to Prometheus/Grafana.

Any specific requests for how the PR should be reviewed?

This is not yet rolled out to the cluster, so eyeball it (and the docs) and do a Task-based rollout while monitoring the effects.

There are some dashboards available which could be imported for easy checking of metrics.

Available metrics are listed here.

[hypesystem]

Nice, rul det gerne ud i platformen og let's get it merged!

[hypesystem]

Du må gerne køre denne ud at your convenience - men sig lige til på Zulip så jeg kan reenable alerts til Zulip :-)

[hypesystem]

This PR was in a bit of an unclear state because we talked about it on Zulip. Basically we'd like to scale up prometheus and loki to be confident in their ability to handle more data ingress before we execute and run this. So it's still waiting a bit until we get over the hump of getting all the libraries live.

[ITViking]

are we in a good enough place to get this in now? @hypesystem and @achton

[achton]

My impression is "yes", but I think you guys are better suited to answer that question.
I've resolved conflicts for ya :-)