Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SBT Parallel execution of tests failing #153

Open
xmariachi opened this issue Aug 1, 2018 · 0 comments
Open

SBT Parallel execution of tests failing #153

xmariachi opened this issue Aug 1, 2018 · 0 comments

Comments

@xmariachi
Copy link

xmariachi commented Aug 1, 2018

I am running tests over the same Kafka Streams topology.
I have different suites extending EmbeddedKafkaStreamsAllInOne.
Each suite uses different ZK and Kafka ports, application name and topic names (same names with appName prefixed).
Each suite contains several tests, which use withRunningKafka and custom consumers, to test different parts of a KafkaStreams application that has several output topics.

When running tests sequentially they're fine, but in parallel starts showing errors of non-existent topics, dangling threads trying to connect to ZK, etc.

I've updated to 1.1.1 to no avail. I've seen this issue has been raised before here, and some fixes have been merged. Are they in 1.1.1 already?

First failure, when executing two suites with appNames "customer-test" and "order-test":

[2018-08-02 13:20:41,446] INFO Create topic: customer-test-sessionz (com.divvit.dp.streams.testing.KafkaStreamsTestUtils$)
[2018-08-02 13:20:41,448] INFO Create topic: order-test-sessionz (com.divvit.dp.streams.testing.KafkaStreamsTestUtils$)
[2018-08-02 13:20:41,452] WARN Error registering AppInfo mbean (org.apache.kafka.common.utils.AppInfoParser)
javax.management.InstanceAlreadyExistsException: kafka.admin.client:type=app-info,id=embedded-kafka-admin-client
	at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
	at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
	at org.apache.kafka.common.utils.AppInfoParser.registerAppInfo(AppInfoParser.java:62)
	at org.apache.kafka.clients.admin.KafkaAdminClient.<init>(KafkaAdminClient.java:380)
	at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:331)
	at org.apache.kafka.clients.admin.AdminClient.create(AdminClient.java:62)
	at net.manub.embeddedkafka.EmbeddedKafkaSupport.createCustomTopic(EmbeddedKafka.scala:716)

I understand here "winner takes it all", in this case "customers" Suite seems to enter first, then "orders" Suite is trying to use the same Kafka but it can't. Is it the client ID: "id=embedded-kafka-admin-client" of any importance here? It is hardcoded here, so I suppose it shouldn't be a matter of conflict.

Then I can find these errors later which seem to show second Suite complaining about its topics:

java.lang.IllegalArgumentException: Assigned partition order-test-customerOrderIntermediate-2 for non-subscribed topic regex pattern; subscription pattern is customer-test-customerOrderIntermediate|customer-test-customerz|customer-test-eventz|customer-test-visitorz
	at org.apache.kafka.clients.consumer.internals.SubscriptionState.assignFromSubscribed(SubscriptionState.java:187)
	at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:220)
	at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:367)
	at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:316)
	at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:290)
	at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1149)
	at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1115)
	at org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:831)
	at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:788)
	at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:749)
	at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:719)

@xmariachi xmariachi changed the title SBT Parallel execution of tests: topics deleted SBT Parallel execution of tests failing Aug 1, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant