impr: propagate VirtualFile metrics via RequestContext

[problame]

Refs

• fixes STORAGE_IO_SIZE metric doesn't use pre-with_label_valued counters #6107

Problem

VirtualFile currently parses the path it is opened with to identify the tenant,shard,timeline labels to be used for the STORAGE_IO_SIZE metric.

Further, for each read or write call to VirtualFile, it uses with_label_values to retrieve the correct metrics object, which under the hood is a global hashmap guarded by a parking_lot mutex.

We perform tens of thousands of reads and writes per second on every pageserver instance; thus, doing the mutex lock + hashmap lookup is wasteful.

Changes

Apply the technque we do for all other timeline-scoped metrics to avoid the repeat with_label_values: add it to TimelineMetrics.

Wrap TimelineMetrics into an Arc.

Propagate the Arc<TimelineMetrics> down do VirtualFile, and use Timeline::metrics::storage_io_size.

We avoid contention on the Arc<TimelineMetrics>'s refcount atomics between different connection handlers for the same timeline.
The technique is to add indirection, using another Arc wrapper.
To avoid frequent allocations, we store that Arc<Arc> inside the per-connection timeline cache.

Preliminary refactorings to enable this change:

• refactor(pageserver): propagate RequestContext to layer downloads #11001
• refactor(page_service / timeline::handle): the GateGuard need not be a special case #11030

Performance

I ran the benchmarks in test_runner/performance/pageserver/pagebench/test_pageserver_max_throughput_getpage_at_latest_lsn.py on an i3en.3xlarge because that's what we currently run them on.

None of the benchmarks shows a meaningful difference in latency or throughput or CPU utilization.

I would have expected some improvement in the many-tenants-one-client-each workload because they all hit that hashmap constantly, and clone the same UintCounter / Arc inside of it.

But apparently the overhead is miniscule compared to the remaining work we do per getpage.

Yet, since the changes are already made, the added complexity is manageable, and the perf overhead of with_label_values demonstable in micro-benchmarks, let's have this change anyway.
Also, propagating TimelineMetrics through RequestContext might come in handy down the line.

Micro-benchmark that demonstrates perf impact of with_label_values, along with other pitfalls and mitigation techniques around the metrics/prometheus crate:

• add benchmark demonstrating metrics/prometheus crate multicore scalability pitfalls & workarounds #11019

Alternative Designs

An earlier iteration of this PR stored an Arc<Arc<Timeline>> inside RequestContext.
The problem is that this risks reference cycles if the RequestContext gets stored in an object that is owned directly or indirectly by Timeline.

Ideally, we wouldn't be using this mess of Arc's at all and propagate Rust references instead.
But tokio requires tasks to be 'static, and so, we wouldn't be able to propagate references across task boundaries, which would be incompatible with fan-out type code.
So, we'd have to be propagating Cow<>s instead.

[github-actions]

7744 tests run: 7366 passed, 0 failed, 378 skipped (full report)

---

Flaky tests (1)

Postgres 17

• test_migration_to_cold_secondary: release-arm64-without-lfc

Code coverage* (full report)

• functions: 32.8% (8655 of 26380 functions)
• lines: 48.6% (73263 of 150671 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
5767947 at 2025-02-27T22:10:58.798Z :recycle:}

[problame]

I extracted a piece of this PR into

• refactor(pageserver): propagate RequestContext to layer downloads #11001

That one I definitely want to land.

[problame]

@arpad-m I was able to remove the blanket Arc<ScopeInner> - we still do the arc_arc for new_timeline(), but no longer for the pagestream case, so, we save one allocation per getpage batch

b6cc0c3