Kafka: Repeated handling of messages after deployment.

[saxap]

We have several instances of the application that consume messages through a group using the jobs plugin + Kafka driver.

During the deployment of the application, we bring up a batch of new consumers, turn off the old ones, and after rebalancing, we start catching repeated processing of messages that were handled 1-3 minutes ago.

I was informed that messages after calling ->ack() first go to the buffered channel and only then are committed to kafka and when we shut down the instance, all messages from this channel do not get read in time and are not marked as commited.

So is this how it works? and is there a way to reduce the number of duplicates after deployment?

[rustatian]

Hey @saxap 👋
Buffered channel does not mean - 'long processing', this is actually a few milliseconds.

[saxap]

Hello @rustatian
Thank you for reply.
So, does it mean messages commited almost instantly after ->ack()? Anyway lag should not be ~1-3 minutes.
It's strange, because before roadrunner we use same kafka broker and consume through rdkafka driver and only sometimes we have 1-3 dublicated handlings. Count of partitions and count consumers is equal and not changed.

[rustatian]

Yeah, offset commited instantly. Lag could be because of various reasons, for example, Kafka rebalancing lags, consumer lags, cluster problems and so on. You may try rdkafka (PHP extension) driver and double-check lags with it. Try to precisely monitor Kafka cluster metrics, logs + RR logs.