Setup replication on production clickhouse instance

[hellais]

There is a scheduled downtime for backend-fsn set for:

Start:	2024-12-05, 03:30 AM UTC
Estimated end:	2024-12-05, 05:30 AM UTC

We should use this as an opportunity and motivation to setup replication between backend-fsn and backend-hel so that we can be fault tolerant to these kinds of downtimes.

Relevant bits of documentation are:

• https://clickhouse.com/docs/en/architecture/replication
• https://clickhouse.com/docs/en/engines/table-engines/mergetree-family/replication#converting-from-mergetree-to-replicatedmergetree

We also need to run a dedicated clickhouse-zookeeper instance in addition to one on the two replicas. It's probably good to set that up on the monitoring host since the memory requirements are relatively light compared to clickhouse.

[hellais]

This is being setup inside of #112 as part of the work to get Pipeline v5 into production. Once it's finished we should move the fastpath clickhouse database over and set it up as a replicated table.

I wrote some notes on this can be done here: https://github.com/ooni/devops/blob/prod-pipeline-v5/docs/merge-tree-replication.md

[hellais]

While we didn't manage to get everything ready for the expected downtime, we did end up deploying a clickhouse cluster which should eventually be used to provide better availability to the backend-fsn host.

This is closed by: #112