Automatic crash dumping and monitoring for "stuck processes"

[ankush]

This is "concepts of a plan", so I don't know specifics but...

We should have some kind of automated core dumping when processes behave badly to do postmortem analysis.

Roughly:

• configure coredumpctl
• Setup some monitoring worker which keeps checking health of all services
• If service isn't healthy for a long time - coredump (and eventually restart)

Why:

• No need to debug under fire, investigate things peacefully.

If not coredumps, we can also do process-specific "dump state" too, like py-spy or sigusr1 for python processes. This doesn't reveal the whole picture though.