Starting profiler for single binary with k8s metrics #3981
Conversation
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
|
This looks reasonable, but CI check is failing. We're retrying in https://github.com/flyteorg/flyte/actions/runs/5944214374?pr=3981. |
| @@ -147,6 +147,20 @@ func startPropeller(ctx context.Context, cfg Propeller) error { | |||
| } | |||
|
|
|||
| if !cfg.DisableWebhook || !cfg.Disabled { | |||
| handlers := map[string]http.Handler{ | |||
| "/k8smetrics": promhttp.HandlerFor(metrics.Registry, promhttp.HandlerOpts{ | |||
There was a problem hiding this comment.
why a separate endpoint? we want to do it so that scraping can be specialized?
There was a problem hiding this comment.
The controller-runtime package automatically populates a prometheus Registry as metrics.Registry. So this was just the easiest way to surface that registry (ie. separate HTTP endpoint). We could wrap both the controller-runtime prometheus registry and our registry (buried in flytestdlib) into a single entity, but it would require a good deal of refactoring. Also it does allowed for specialized scraping too.
There was a problem hiding this comment.
Is there any advantage to unifying the registries?
There was a problem hiding this comment.
@eapolinario AFAIK just that there would be a single endpoint to retrieve all metrics.
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
* starting profile for single binary Signed-off-by: Daniel Rammer <[email protected]> * updating datacatalog Signed-off-by: Daniel Rammer <[email protected]> --------- Signed-off-by: Daniel Rammer <[email protected]> Signed-off-by: Eduardo Apolinario <[email protected]>
Tracking issue
#2975
Describe your changes
Starting the profiler http endpoint for the single binary and exposing k8s metrics using a separate handler.
Check all the applicable boxes
Screenshots
NA
Note to reviewers
NA