Received duplicated records when rebalance occurs #360
-
Description of my configuration for parallel consumers:
I have created an integration test to push events with specific sequence and during the consuming and sending events to a testing endpoint, is simulated a rebalance process simply connect another kafka consumer with the same groupId and after a while, disconnecting and closing the new kafka consumer. |
Beta Was this translation helpful? Give feedback.
Replies: 5 comments 11 replies
-
Hi, welcome to the project! Are you able to write the test as an integration test in the vertx module? Here is the code that commits on revocation: Ignore the comment on line #342, it's wrong and still commits for revoked partitions. It should say "lost" not "revoked"... Hmm, I do notice that in commit #2aa4da956d6b8b49a09472e75a92c9e1023d1e4a, the order of work truncation and commit was reversed, which makes me suspicious.. |
Beta Was this translation helpful? Give feedback.
-
Could you compile for me the jar for java 8 target and send to me? I have lots of error while creating the jar from your branch (truth-generator-maven- plug in could not be resolved). Thank
|
Beta Was this translation helpful? Give feedback.
-
Here the command I run and the logs:
```
avitale-mac:parallel-consumer avitale$ mvn clean install -U
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO]
[INFO] Confluent Parallel Consumer [pom]
[INFO] Confluent Parallel Consumer Core [jar]
[INFO] Confluent Parallel Consumer Vert.x [jar]
[INFO] Confluent Parallel Consumer Project Reactor.io [jar]
[INFO] Confluent Parallel Consumer Examples [pom]
[INFO] Confluent Parallel Consumer Example - Core [jar]
[INFO] Confluent Parallel Consumer Example - Streams [jar]
[INFO] Confluent Parallel Consumer Example - Vert.x [jar]
[INFO] Confluent Parallel Consumer Example - Project Reactor.io [jar]
[INFO]
[INFO] -------< io.confluent.parallelconsumer:parallel-consumer-parent >-------
[INFO] Building Confluent Parallel Consumer 0.5.3.0-SNAPSHOT [1/9]
[INFO] --------------------------------[ pom ]---------------------------------
[INFO]
[INFO] --- maven-clean-plugin:3.2.0:clean (default-clean) @ parallel-consumer-parent ---
[INFO] Deleting /Users/avitale/repo/parallel-consumer/target
[INFO]
[INFO] --- ossindex-maven-plugin:3.2.0:audit (audit-dependencies) @ parallel-consumer-parent ---
[INFO] Skipping; POM module
[INFO]
[INFO] --- maven-enforcer-plugin:3.1.0:enforce (enforce-versions) @ parallel-consumer-parent ---
[INFO]
[INFO] --- jacoco-maven-plugin:0.8.8:prepare-agent (default) @ parallel-consumer-parent ---
[INFO] argLine set to -javaagent:/Users/avitale/.m2/repository/org/jacoco/org.jacoco.agent/0.8.8/org.jacoco.agent-0.8.8-runtime.jar=destfile=/Users/avitale/repo/parallel-consumer/target/jacoco.exec
[INFO]
[INFO] --- maven-dependency-plugin:3.3.0:tree (default) @ parallel-consumer-parent ---
[INFO] io.confluent.parallelconsumer:parallel-consumer-parent:pom:0.5.3.0-SNAPSHOT
[INFO] +- org.projectlombok:lombok:jar:1.18.24:provided
[INFO] +- org.slf4j:slf4j-api:jar:1.7.36:compile
[INFO] +- pl.tlinkowski.unij:pl.tlinkowski.unij.api:jar:0.1.3:compile
[INFO] | \- pl.tlinkowski.unij:pl.tlinkowski.unij.service.api:jar:0.1.3:runtime
[INFO] +- pl.tlinkowski.unij:pl.tlinkowski.unij.bundle.jdk8:jar:0.1.3:compile
[INFO] | +- pl.tlinkowski.unij:pl.tlinkowski.unij.service.collect.jdk8:jar:0.1.3:runtime
[INFO] | \- pl.tlinkowski.unij:pl.tlinkowski.unij.service.misc.jdk8:jar:0.1.3:runtime
[INFO] +- ch.qos.logback:logback-classic:jar:1.2.11:test
[INFO] | \- ch.qos.logback:logback-core:jar:1.2.11:test
[INFO] +- org.junit.jupiter:junit-jupiter-api:jar:5.8.2:test
[INFO] | +- org.opentest4j:opentest4j:jar:1.2.0:test
[INFO] | \- org.apiguardian:apiguardian-api:jar:1.1.2:test
[INFO] +- org.junit.jupiter:junit-jupiter-engine:jar:5.8.2:test
[INFO] | \- org.junit.platform:junit-platform-engine:jar:1.8.2:test
[INFO] +- org.junit.platform:junit-platform-launcher:jar:1.8.2:test
[INFO] +- org.junit.platform:junit-platform-commons:jar:1.8.2:test
[INFO] +- org.assertj:assertj-core:jar:3.23.1:test
[INFO] | \- net.bytebuddy:byte-buddy:jar:1.12.10:provided
[INFO] +- org.mockito:mockito-core:jar:4.6.1:test
[INFO] | +- net.bytebuddy:byte-buddy-agent:jar:1.12.10:provided
[INFO] | \- org.objenesis:objenesis:jar:3.2:test
[INFO] +- org.mockito:mockito-junit-jupiter:jar:4.6.1:compile
[INFO] +- com.google.auto.service:auto-service-annotations:jar:1.0.1:test
[INFO] +- com.google.truth:truth:jar:1.1.3:test
[INFO] | +- com.google.guava:guava:jar:31.1-jre:test
[INFO] | | +- com.google.guava:failureaccess:jar:1.0.1:test
[INFO] | | +- com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava:test
[INFO] | | +- com.google.code.findbugs:jsr305:jar:3.0.2:test
[INFO] | | \- com.google.j2objc:j2objc-annotations:jar:1.3:test
[INFO] | +- org.checkerframework:checker-qual:jar:3.13.0:test
[INFO] | +- junit:junit:jar:4.13.2:test
[INFO] | | \- org.hamcrest:hamcrest-core:jar:1.3:test
[INFO] | +- com.google.auto.value:auto-value-annotations:jar:1.8.1:test
[INFO] | +- com.google.errorprone:error_prone_annotations:jar:2.7.1:test
[INFO] | \- org.ow2.asm:asm:jar:9.1:test
[INFO] +- com.google.truth.extensions:truth-java8-extension:jar:1.1.3:test
[INFO] +- org.awaitility:awaitility:jar:4.2.0:test
[INFO] | \- org.hamcrest:hamcrest:jar:2.1:test
[INFO] +- com.google.flogger:flogger:jar:0.7.4:test
[INFO] | \- org.checkerframework:checker-compat-qual:jar:2.5.3:test
[INFO] +- com.google.flogger:flogger-slf4j-backend:jar:0.7.4:test
[INFO] | \- com.google.flogger:flogger-system-backend:jar:0.7.4:test
[INFO] +- one.util:streamex:jar:0.8.1:test
[INFO] \- com.github.bsideup.jabel:jabel-javac-plugin:jar:0.4.2:provided
[INFO] \- net.java.dev.jna:jna:jar:5.9.0:provided
[INFO]
[INFO] --- lombok-maven-plugin:1.18.20.0:delombok (default) @ parallel-consumer-parent ---
[WARNING] Skipping Delombok; no source to process.
[INFO]
[INFO] --- asciidoc-template-maven-plugin:1.0.20:build (process) @ parallel-consumer-parent ---
[INFO]
[INFO] --- license-maven-plugin:4.2.rc3:format (default) @ parallel-consumer-parent ---
[INFO] Updating license headers...
[INFO]
[INFO] --- maven-help-plugin:3.2.0:active-profiles (show-profiles) @ parallel-consumer-parent ---
[INFO]
Active Profiles for Project 'io.confluent.parallelconsumer:parallel-consumer-parent:pom:0.5.3.0-SNAPSHOT':
The following profiles are active:
- artifactory (source: external)
- license-format (source: io.confluent.parallelconsumer:parallel-consumer-parent:0.5.3.0-SNAPSHOT)
Active Profiles for Project 'io.confluent.parallelconsumer:parallel-consumer-core:jar:0.5.3.0-SNAPSHOT':
The following profiles are active:
- artifactory (source: external)
- license-format (source: io.confluent.parallelconsumer:parallel-consumer-parent:0.5.3.0-SNAPSHOT)
Active Profiles for Project 'io.confluent.parallelconsumer:parallel-consumer-vertx:jar:0.5.3.0-SNAPSHOT':
The following profiles are active:
- artifactory (source: external)
- license-format (source: io.confluent.parallelconsumer:parallel-consumer-parent:0.5.3.0-SNAPSHOT)
Active Profiles for Project 'io.confluent.parallelconsumer:parallel-consumer-reactor:jar:0.5.3.0-SNAPSHOT':
The following profiles are active:
- artifactory (source: external)
- license-format (source: io.confluent.parallelconsumer:parallel-consumer-parent:0.5.3.0-SNAPSHOT)
Active Profiles for Project 'io.confluent.parallelconsumer:parallel-consumer-examples:pom:0.5.3.0-SNAPSHOT':
The following profiles are active:
- artifactory (source: external)
- license-format (source: io.confluent.parallelconsumer:parallel-consumer-parent:0.5.3.0-SNAPSHOT)
Active Profiles for Project 'io.confluent.parallelconsumer:parallel-consumer-example-core:jar:0.5.3.0-SNAPSHOT':
The following profiles are active:
- artifactory (source: external)
- license-format (source: io.confluent.parallelconsumer:parallel-consumer-parent:0.5.3.0-SNAPSHOT)
Active Profiles for Project 'io.confluent.parallelconsumer:parallel-consumer-example-streams:jar:0.5.3.0-SNAPSHOT':
The following profiles are active:
- artifactory (source: external)
- license-format (source: io.confluent.parallelconsumer:parallel-consumer-parent:0.5.3.0-SNAPSHOT)
Active Profiles for Project 'io.confluent.parallelconsumer:parallel-consumer-example-vertx:jar:0.5.3.0-SNAPSHOT':
The following profiles are active:
- artifactory (source: external)
- license-format (source: io.confluent.parallelconsumer:parallel-consumer-parent:0.5.3.0-SNAPSHOT)
Active Profiles for Project 'io.confluent.parallelconsumer:parallel-consumer-example-reactor:jar:0.5.3.0-SNAPSHOT':
The following profiles are active:
- artifactory (source: external)
- license-format (source: io.confluent.parallelconsumer:parallel-consumer-parent:0.5.3.0-SNAPSHOT)
[INFO]
[INFO] --- build-helper-maven-plugin:3.3.0:add-test-source (add-integration-test-source) @ parallel-consumer-parent ---
[INFO] Test Source directory: /Users/avitale/repo/parallel-consumer/src/test-integration/java added.
[INFO] Test Source directory: /Users/avitale/repo/parallel-consumer/target/generated-sources added.
[INFO]
[INFO] --- build-helper-maven-plugin:3.3.0:add-test-resource (add-integration-test-resource) @ parallel-consumer-parent ---
[INFO]
[INFO] --- jacoco-maven-plugin:0.8.8:report (report) @ parallel-consumer-parent ---
[INFO] Skipping JaCoCo execution due to missing execution data file.
[INFO]
[INFO] --- maven-jar-plugin:3.2.2:test-jar (default) @ parallel-consumer-parent ---
[WARNING] JAR will be empty - no content was marked for inclusion!
[INFO] Building jar: /Users/avitale/repo/parallel-consumer/target/parallel-consumer-parent-0.5.3.0-SNAPSHOT-tests.jar
[INFO]
[INFO] --- maven-javadoc-plugin:3.4.0:jar (attach-javadocs) @ parallel-consumer-parent ---
[INFO] Not executing Javadoc as the project is not a Java classpath-capable package
[INFO]
[INFO] --- maven-source-plugin:3.2.1:jar-no-fork (attach-sources) @ parallel-consumer-parent ---
[INFO]
[INFO] --- maven-failsafe-plugin:3.0.0-M7:integration-test (default) @ parallel-consumer-parent ---
[INFO] No tests to run.
[INFO]
[INFO] --- maven-failsafe-plugin:3.0.0-M7:verify (default) @ parallel-consumer-parent ---
[INFO]
[INFO] --- maven-install-plugin:3.0.0-M1:install (default-install) @ parallel-consumer-parent ---
[INFO] Installing /Users/avitale/repo/parallel-consumer/pom.xml to /Users/avitale/.m2/repository/io/confluent/parallelconsumer/parallel-consumer-parent/0.5.3.0-SNAPSHOT/parallel-consumer-parent-0.5.3.0-SNAPSHOT.pom
[INFO] Installing /Users/avitale/repo/parallel-consumer/target/parallel-consumer-parent-0.5.3.0-SNAPSHOT-tests.jar to /Users/avitale/.m2/repository/io/confluent/parallelconsumer/parallel-consumer-parent/0.5.3.0-SNAPSHOT/parallel-consumer-parent-0.5.3.0-SNAPSHOT-tests.jar
[INFO]
[INFO] --------< io.confluent.parallelconsumer:parallel-consumer-core >--------
[INFO] Building Confluent Parallel Consumer Core 0.5.3.0-SNAPSHOT [2/9]
[INFO] --------------------------------[ jar ]---------------------------------
Downloading from liveperson: https://artifactory.int.liveperson.net/artifactory/liveperson/io/stubbs/truth/truth-generator-maven-plugin/0.1.1/truth-generator-maven-plugin-0.1.1.pom
[WARNING] The POM for io.stubbs.truth:truth-generator-maven-plugin:jar:0.1.1 is missing, no dependency information available
Downloading from liveperson: https://artifactory.int.liveperson.net/artifactory/liveperson/io/stubbs/truth/truth-generator-maven-plugin/0.1.1/truth-generator-maven-plugin-0.1.1.jar
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Confluent Parallel Consumer 0.5.3.0-SNAPSHOT:
[INFO]
[INFO] Confluent Parallel Consumer ........................ SUCCESS [ 5.614 s]
[INFO] Confluent Parallel Consumer Core ................... FAILURE [ 48.644 s]
[INFO] Confluent Parallel Consumer Vert.x ................. SKIPPED
[INFO] Confluent Parallel Consumer Project Reactor.io ..... SKIPPED
[INFO] Confluent Parallel Consumer Examples ............... SKIPPED
[INFO] Confluent Parallel Consumer Example - Core ......... SKIPPED
[INFO] Confluent Parallel Consumer Example - Streams ...... SKIPPED
[INFO] Confluent Parallel Consumer Example - Vert.x ....... SKIPPED
[INFO] Confluent Parallel Consumer Example - Project Reactor.io SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 54.392 s
[INFO] Finished at: 2022-07-25T23:50:15+02:00
[INFO] ------------------------------------------------------------------------
[ERROR] Plugin io.stubbs.truth:truth-generator-maven-plugin:0.1.1 or one of its dependencies could not be resolved: Could not find artifact io.stubbs.truth:truth-generator-maven-plugin:jar:0.1.1 in liveperson (https://artifactory.int.liveperson.net/artifactory/liveperson) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginResolutionException
```
|
Beta Was this translation helpful? Give feedback.
-
Please send me the jar compiled from your branch. Thanks
|
Beta Was this translation helpful? Give feedback.
-
Fix and verifying test pushed here: #365 |
Beta Was this translation helpful? Give feedback.
Fix and verifying test pushed here: #365