GPU-related prometheus metrics

[mrocklin]

If GPUs are present we have some nice Dask dashboards that give us real-time information about things like GPU memory and GPU utilization.

It would be nice to expose these also as prometheus metrics for offline analysis.

cc @jacobtomlinson @crusaderky @ntabris

[ntabris]

Is there a benefit to exposing these via Dask, rather than expecting folks to use https://github.com/NVIDIA/dcgm-exporter if they want GPU metrics? Does Dask have distinct GPU-related metrics? (Genuine question, I'm not sure.)

[mrocklin]

Not particularly distinct. If there is some standard for this already that people can use probably that's fine. I'd defer to @jacobtomlinson though

[jacobtomlinson]

The kind of fine memory metrics that @charlesbluca is talking about in #8148 wouldn't be exposed by DCGM so there probably is value in exposing that in Dask.