safekeeper: block deletion on protocol handler shutdown

[jcsp]

Problem

Two recently observed log errors indicate safekeeper tasks for a timeline running after that timeline's deletion has started.

• safekeeper: WAL removal task failed: No such file or directory #8972
• safekeeper: root-cause WAL sender: failed to open WAL file #8974

These code paths do not have a mechanism that coordinates task shutdown with the overall shutdown of the timeline.

Summary of changes

• Add a Gate to Timeline
• Take the gate as part of resident timeline guard: any code that holds a guard over a timeline staying resident should also hold a guard over the timeline's total lifetime.
• Take the gate from the wal removal task
• Respect Timeline::cancel in WAL send/recv code, so that we do not block shutdown indefinitely.
• Add a test that deletes timelines with open pageserver+compute connections, to check these get torn down as expected.

There is some risk to introducing gates: if there is code holding a gate which does not properly respect a cancellation token, it can cause shutdown hangs. The risk of this for safekeepers is lower in practice than it is for other services, because in a healthy timeline deletion, the compute is shutdown first, then the timeline is deleted on the pageserver, and finally it is deleted on the safekeepers -- that makes it much less likely that some protocol handler will still be running.

Closes: #8972
Closes: #8974

Checklist before requesting a review

• I have performed a self-review of my code.
• If it is a core feature, I have added thorough tests.
• Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
• If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

• Do not forget to reformat commit message to not include the above checklist

[github-actions]

5534 tests run: 5308 passed, 0 failed, 226 skipped (full report)

---

Flaky tests (1)

Postgres 17

• test_idle_checkpoints: debug-x86-64

Code coverage* (full report)

• functions: 31.5% (7935 of 25227 functions)
• lines: 49.6% (62952 of 126943 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
96ae2bb at 2024-11-20T10:33:23.074Z :recycle:}

[jcsp]

Addressed some comments & rebased on main, a few still changes are still oustanding

[arssher]

One more note: it would be reasonable to add GateGuard to Manager as well, otherwise deletion doesn't wait for manager exit.

[jcsp]

One more note: it would be reasonable to add GateGuard to Manager as well, otherwise deletion doesn't wait for manager exit.

Added GateGuard in the task where we run Manager in ba40e2c

[jcsp]

Sorry for the high-latency turnaround. I've been through all the comments now, although haven't done a full re-self-review to make sure it all still makes sense.

[erikgrinaker]

We should move the send/receive WAL cancellation up to the task level and just drop the futures, otherwise this LGTM.

Much simpler this time around, appreciate the cleanup.

[erikgrinaker]

Thanks, this cleaned up pretty nicely.

[arssher]

LGTM except +1's on Eric's comments.