Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to support at least once for CamelKafkasourceSourceConnector #1639

Open
seanshi007 opened this issue May 17, 2024 · 0 comments
Open

How to support at least once for CamelKafkasourceSourceConnector #1639

seanshi007 opened this issue May 17, 2024 · 0 comments

Comments

@seanshi007
Copy link

We've been testing a few Camel Kafka Connectors (more specifically PubSub source connector and Kafka source connector). And it looks like PubSub source connector is able to support at least once by ack the message via calling the oncompletion synchronization in the override commitRecord method.

However, for Kafka source connector, even if we disable auto commit, enable manual commit, create a new connector and task that extends the auto generated CamelKafkasourceSourceConnector and CamelKafkasourceSourceTask, and override the commitRecord in the extended task to get the KafkaManualCommit from the exchange's header and try to call it to commit, because the actual Kafka consumer and connector task are two different threads, the commit is never going to work as Kafka consumer can only be operated on the same thread. So, I am just wondering is there any approach that we can make the Camel Kafka source connector support at least once?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant