PAUSE MIRROR support

[heavycrystal]

PAUSE MIRROR supports both CDC and QRep mirrors. This is implemented by signalling the Temporal workflow. A PauseSignal delivered to the workflow should pause it, while a NoopSignal delivered to the workflow should resume it. Signals that don't make sense for the current workflow state are ignored. Currently supported only via the query layer, not from the UI. The following caveats exist for now:

1. QRep mirrors cannot be paused during partition reading or replicating, only after a run completes successfully and the signals are read. This is usually not an issue but could be problematic for QRep runs with a lot of partitions are running.
2. CDC mirrors similarly will only process signals and pause after a pull + sync + normalize cycle is run. Due to 1), a CDC mirror cannot pause during the initial snapshot phase, which is implemented as a QRep mirror internally. This will be marked as unpausable in the UI.

[serprex]

So PauseSignal triggers flipping pause state? Seems like a source of flakiness compared to having explicit Pause/Unpause signal. But then you'd have to consider what to do when both signals are set. Are signals not able to carry boolean data?

Guess it also doesn't matter since handler manages sequencing, so you handle pause/unpause in order received

[heavycrystal]

I agree that the current logic is a little flaky, but at the same time I can't think of an obvious way it will fail, since we only read a signal at a time and we only maintain one state variable. Temporal Signals are able to carry serializable data, but carrying a boolean value on the Pause signal isn't very clear IMO.

We could also use explicit Pause and Unpause signals and ignore an illegal one [pausing when already paused] but this would require two RPC endpoints exposed. Another consideration is that the UI can prevent illegal state changes by reading the current state of the workflow, but our query layer has no such ability.

[serprex]

In the minute while it's asleep one can imagine someone sending redundant pause signals

[serprex]

is there some way to share this shutdown/pause logic?

Also seems like activeSignal shouldn't be switched from ShutdownSignal to PauseSignal. So should be checking activeSignal to prevent that sort of state transition

[heavycrystal]

The data structures associated with CDCFlow and QRepFlow are different enough for sharing the logic to be not trivial.

I don't think there's a way to transition from ShutdownSignal to PauseSignal, as we only process one signal at a time and the return from shutdown happens immediately.

[serprex]

Could have utility function in shared that takes signal value + active signal value & returns new active signal value