Kip-848: Next Generation Consumer Group

[aratz-lasa]

I wanted to know what is the plan on implementing KIP-848. I see a dedicated branch just for this, but it's been inactive for a year. @twmb I would be happy to help move it forward.

[twmb]

I have to evaluate what this entails. The branch is stale because I used to implement protocol support before it was rolled out in Kafka, but stopped once Kafka started changing APIs repeatedly before the API was finally released.

I'm not sure 848 is high priority though because it's released as preview only. I'm not sure how hard it'll be to implement, but likely pretty hard...

[aratz-lasa]

Yeah, it's pretty hard. I've been reading all the JIRA tasks they have, and it's crazy. They made significant changes to the client. Here you can see notes about the new client-side state machine: https://cwiki.apache.org/confluence/display/KAFKA/Consumer+rebalance#Consumerrebalance-RebalanceStateMachine

Most of the remaining tasks will be merged for 3.8.0 release (which will be in May). However, I didn't find any official source on the timeline.

[twmb]

Yeah I'm probably going to need help with this, this is a big one.

Otherwise this may be unsupported for a while while I slowly slowly tinker away adding support. It seems like a lot of work for minor gain.

[aratz-lasa]

@twmb I'm able to help you. This week, I'm starting to implement the server-side logic, so I'll get a good understanding of the topic. If you want we can collaborate on this work.

[twmb]

I definitely missed your reply above^.

I'm starting to think about this (a little bit). One thing I may lean to do first is to improve the guts of the consumer so that moving a partition between brokers does not require a full consumer stop/restart -- and then after, implement KIP-848.

Or it may make sense to just start with 848 logic, I'm really not sure -- I haven't dug into it much.