Few consumers not consuming messages from partitions

[vaibhavg1985apr]

Hello,

We have set 40 partitions and created 40 consumer threads. However, only 36 consumers are able to consume messages rest 4 consumers are not getting any messages. Could you please know in what scenario this can be possible and how to resolve that. Is there a way, we can manually link partition to consumer thread?

[treziac]

Rdkafka-dotnet is not maintained anymore, please move to https://github.com/confluentinc/confluent-kafka-dotnet which replace it

With rdkafka or confluent Kafka, you should not create 40 consumers in the same process - create a single one which subscribe to whole topics, and dispatch message to workers thread if needed (via concurrent queue for example), or dispatch to multiple process machines (you should not observe this though, so feel free to post more details in confluent repo!)

[vaibhavg1985apr]

Hello.

I haven't created 40 consumers in same process. We have 40 instances with same consumer group each having one thread for consuming messages from Kafka. Patterns we are seeing are not constant. Current issue we see that out of 40 consumers only 36 consumers are able to map with partition and consuming messages. Do you have any idea how this could be possible and what needs to be done to fix this issue.

[treziac]

Can you please specify if you are using rdkafka or confluent-kafka and version&os&some code snippets? Thks!
Also won't be able to help for now, going off to we - will answer more throughfully on Monday. You can search on librdkafka or confluent-kafka if problem already occured. Setting debug config to "cgrp" and providing logs will help a lot :)

[vaibhavg1985apr]