Shuffle metrics 3/4: Capture background metrics

[crusaderky]

• Closes Use metering for P2P shuffling instrumentation #7943

Please read: #7943 (comment)
There are three commits in this PR. All but the last are the previous PRs in the chain.

Design note

There were two orthogonal ways to deliver this.

The first one (which I tried and scrapped) was collecting the metrics locally in a ShuffleRun.metrics attribute and then used the heartbeat system to propagate them. ShuffleSchedulerPlugin.heartbeat became substantially more complicated, as it had to propagate the data locally to Scheduler.cumulative_worker_metrics as well as to the spans extension. The biggest flaw was that it was completely losing the metrics generated between a heartbeat and the shuffle being deleted on the scheduler side. In order to remediate I would have had to implement some sort of shuffle_id -> span_id mapping for recently deleted runs.

The second design is what you see in this PR. It feels to me a bit more coherent with what we already have (e.g. how Worker.rpc is already passed to the ShuffleRun).

TBH both designs feel a bit hacky in their own way. I don't have a strong preference for either.

[github-actions]

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

       27 files  ±  0         27 suites  ±0   11h 55m 31s ⏱️ + 17m 15s
  3 948 tests +  4    3 833 ✔️  -   2     110 💤 +1    5 ❌ +  5 
49 659 runs  +37  47 344 ✔️ +19  2 301 💤 +4  14 ❌ +14 

For more details on these failures, see this check.

Results for commit bc7e57f. ± Comparison against base commit 4d41d32.

♻️ This comment has been updated with latest results.

[hendrikmakait]

Thanks, @crusaderky!