New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Kafka重复消费问题 #134

Open

cobolbaby opened this issue Mar 26, 2021 · 0 comments

cobolbaby commented Mar 26, 2021 •

edited

Loading

如果Transformer/Merger/Loader线程执行较慢，那Kafka Offset就迟迟不能提交。

这种情况下，应该会造成重复消费到同样的数据吧。具体见如下代码:

bireme/src/main/java/cn/hashdata/bireme/pipeline/KafkaPipeLine.java

Lines 44 to 51 in 9cfc128

    
           public ChangeSet pollChangeSet() throws BiremeException { 
        
             ConsumerRecords<String, String> records = null; 
        
             try { 
        
               records = consumer.poll(POLL_TIMEOUT); 
        
             } catch (InterruptException e) { 
        
             }

Ps: 看来bireme只能依赖主键做兜底了。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment