Emit metrics related to how many tablets are owned by a CN #26
Conversation
| public static final MetricWithLabelGroup<LongCounterMetric> COUNTER_CN_SELECTED_FOR_TABLET_SCAN = | ||
| new MetricWithLabelGroup<>("cn_id", | ||
| () -> new LongCounterMetric("cn_selected_for_tablet_scan", MetricUnit.NOUNIT, | ||
| "times this FE selected the given CN for a scan of some tablet")); |
There was a problem hiding this comment.
Would the correct way to interpret/use this metric be something like:
(metric at t)-(metric at t-1) is how many times an FE selected a particular CN node to scan any tablet in the past minute?
If (metric at t)-(metric at t-1) for a CN node 1 is greater than (metric at t)-(metric at t-1) for a CN node 2, then CN node 1 is hotter than node 2 given most other variables are held the same like total tablet size scanned.
There was a problem hiding this comment.
Yes, exactly. You've hit on part of its utility -- if this metric's value is relatively similar for all CN, but one CN has higher CPU util, we have an indication that some tablets are likely too big, or receiving too much ingestion, or the CN is otherwise doing some other work.
Why I'm doing:
Want to emit these to dashboards so we can be relatively sure that tablets are well balanced across CN and no CN is too hot based on load assigned.
What I'm doing:
Fixes #issue
What type of PR is this:
Does this PR entail a change in behavior?
If yes, please specify the type of change:
Checklist:
Bugfix cherry-pick branch check: