storcon: separate drivers for pageserver and safekeeper heartbeats

[VladLazar]

The storage controller heartbeats al pageservers and safekeepers in a region.
The heartbeats are sent concurrently here, but the handling code below waits
for both futures to complete.

This means that handling of pageserver heartbeats can stall on safekeeper heartbeats
and vice-versa. Currently, some things rely on up-to date pageserver node availability
(e.g. transition from warming up state to active block filling the node after restart).

There's a time-out in place for the safekeeper heartbeats, but since that's likely not a long
term solution we should have separate heartbeat drivers. The cost is a new tokio task that's
idle for most of the time.

[arpad-m]

Personally I don't see much benefit in having two heartbeat drivers, because in the short term we have the timeouts as you say, and in the long term, we'll have downtime anyways when one of the heartbeat drivers is down: timeline creation and deletion will depend on safekeepers. But it is hopefully not hard to implement, and it definitely doesn't hurt to do it.

[VladLazar]

we'll have downtime anyways when one of the heartbeat drivers is down: timeline creation and deletion will depend on safekeepers.

Sure, timeline operation downtime is one way of looking at it. Not the only way though, delaying PS heartbeat handling causes other operational pains like we've seen in https://github.com/neondatabase/cloud/issues/24396.

[arpad-m]

Fair point, I originally thought that https://github.com/neondatabase/cloud/issues/24396 was about the init phase but turns out it was about the post-init phase and has nothing to do with timeline creation/deletion. So there it's definitely relevant, and two drivers would indeed give a net increase in robustness.