Consumer backoff strategy

[klippx]

Let's say I have a message that fails, I want to retry it 5 times with an exponential backoff strategy before giving up and returning an error. It doesn't look like there is any way to achieve this with this library, or how would you do that?

When I search for an answer I see the closest thing that makes sense in consumer API config would be Consumer.Retry.BackoffFunc, but no, it seems like all it is trying to achieve is retrying reading from a partition. I need something that is about retrying a failed message (how many times and with what backoff).

A message can fail due to temporary glitches in the upstreams I am calling or they are returning 429 and I need to wait a bit before retrying again. Surely this is one of the most basic issues a consumer might face and there must be a way?

[dnwe]

I need something that is about retrying a failed message

I think what you're saying is that in your Sarama consumer you're reading messages from the subscribed topic partitions and then making HTTP requests to an upstream containing the data from your consumed messages and you want to retry those attempts several times before giving up on that record and consuming further messages from Kafka?

[klippx]

Hi @dnwe!

What you describe there would be 100% userland code which I am in control over.

But in this case I am wondering if you have a built in retry mechanism for failed messages in general:

1. Message received: Message fails to process
2. Retry once after 0.5s: Message fails again
3. Retry again after 2s: Message fails again
4. ...
5. Last attempt: Message fails again
6. Give up and throw / return error.

Something like this: tulios/kafkajs#1473 But something idiomatic or built in to/offered by the Sarama library.

[dnwe]

It is the "Message fails to process" bit that I don't understand here. From Sarama's perspective a message body is received by the consumer from a FetchRequest to Kafka or it is not, and then it is handed over to the user code to process it.

The alternatives are the FetchRequest to Kafka has an empty reply (because there's no new messages) or Kafka returns an error code in the FetchResponse and Sarama will handle that error as appropriate (e.g., reconnecting / refreshing metadata etc.) and retry the Fetch operation

[klippx]

Ok, I realize this is not a problem that this package intends to solve.

To move on I solved it by wrapping each handler with a Exponential Backoff package, this way if a handler throws it will retry automatically a configurable amount of times using a configurable backoff strategy.