util/metric: NetworkLatencyBuckets ceiling too low

[erikgrinaker]

The NetworkLatencyBuckets histogram buckets max out at 1 second. This is too low for most of the current users, where latencies can easily be several seconds when clusters are struggling, and we'd want to see the actual latencies -- in particular:

• liveness.heartbeatlatency
• leases.requests.latency
• kv.prober.read.latency
• kv.prober.write.latency

@dhartunian What would you recommend for these? Should we change them to IOLatencyBuckets, or increase the range on NetworkLatencyBuckets to 10 seconds? Is it safe to do so from a backwards compatibility perspective?

Jira issue: CRDB-28314

[ericharmeling]

Should we change them to IOLatencyBuckets

This option looks to be the least disruptive, as I think we want to avoid losing fidelity in the histograms for which the current NetworkLatencyBuckets fit well.

#97144 tracks finding a more data-driven solution to fitting bucket sizes and limits to individual metric histograms.

[erikgrinaker]

Yeah, we can do that. Will it cause any problems to simply flip them?

I think we want to avoid losing fidelity in the histograms for which the current NetworkLatencyBuckets fit well.

I think the problem is that the current buckets aren't appropriate for any metrics. Latencies can easily exceed a second when clusters are struggling, and we need to capture that.

[ericharmeling]

Will it cause any problems to simply flip them?

AFAIK, these buckets are only used in the calculation of the quantile values stored in tsdb, so we should be good to change the buckets used and then backport (noting that there will be some expected changes in the p-values recorded following an upgrade).

I think the problem is that the current buckets aren't appropriate for any metrics. Latencies can easily exceed a second when clusters are struggling, and we need to capture that.

A cursory look at the corresponding prometheus metrics in centmon supports your claim. In addition to the metrics you listed in this issue description, looks like we are really only using NetworkLatencyBuckets for proxy.conn_migration.attempted.latency.

I'm in favor of switching all over to IOLatencyBuckets, especially given the fact that the bucket size increases logarithmically, retaining a good amount of fidelity at the lower end. I'll open a PR.

[dhartunian]

AFAIK, these buckets are only used in the calculation of the quantile values stored in tsdb, so we should be good to change the buckets used and then backport (noting that there will be some expected changes in the p-values recorded following an upgrade).

@ericharmeling these buckets are also scraped by any prometheus instance we or the customer uses directly via _status/vars. I think in general changing the bucket boundaries is acceptable especially for increased accuracy because customers are also typically computing percentiles from them as well.