Transactional Pipeline #2309
Closed
chubei
started this conversation in
Feature Requests
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Transactional Pipeline
The pipeline has
ExecutorOperation::Commit
now, but the commit message is generated by theSourceChannelManager
, which tracks the number of uncommited operations and the time elapsed since the last commit. As a result, the commit does not necessarily align with source transaction boundaries.In this proposal, we introduce
IngestionMessage::Commit
, which should be sent from the connector at the end of each source transaction. For the sources that do not support transactions, they should sendIngestionMessage::Commit
periodically, typically before beginning polling for new data.Connector Example
Postgres
Postgres supports transactions, so in the replication phase. Postgres connector should send
IngestionMessage::Commit
when a transaction ends. In the snapshotting phase, Postgres connector should not sendIngestionMessage::Commit
.Object store
Object store connector can send
IngestionMessage::Commit
after every file is ingested.Beta Was this translation helpful? Give feedback.
All reactions