Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Clickhouse: add select consistency (#2300)
### Overview [A recent effort ](https://github.com/PeerDB-io/peerdb/pull/2256/files) was made to improve PeerDB's ingestion performance into ClickHouse by parallelizing the INSERT INTO SELECT queries which took data from the raw table and put it into the final tables. Just before this step, PeerDB moves data from S3 to the raw table. ### Problem In this implementation, every insert runs on a new ClickHouse session - which means the inserts could be running on different nodes/replicas. In this case, there exists a non-zero chance (albeit very unliikely) that the INSERT INTO SELECT does not read the rows in the raw table inserted by the first step. This is documented here: https://clickhouse.com/docs/en/cloud/reference/shared-merge-tree#consistency ### Solution This PR adds the setting `select_sequential_consistency = 1` in the `Connect` function which we use for connecting to ClickHouse
- Loading branch information