Introduction blog post , precise "more efficient"

[raphaelauv]

Hi in the blog post : https://www.confluent.io/blog/introducing-confluent-parallel-message-processing-client/

Say, a 50-partition topic might be processed using five instances of the Parallel Consumer, each running on separate machines (i.e., using Kafka’s consumer group feature where the number of consumers in the group is five). Each instance would process messages using multiple threads giving a total parallelism of 50. This is more efficient than using a consumer group of 50 consumer instances.

I think say more efficient is a little too simple, the article could develop the different trade-off (advantage and inconvenient ) :

• on CPU/RAM consumption by the consuming application/deployment
• the general latency of consumption depending of the kafka cluster configuration ( brokers ).
• more consumers mean less latency but more consumer-group synchronization cost ...

Theses are the first things I can think of.

Also some benchmarks demonstrating the advantages of parallel-consumer would be very nice.

Thank you

[astubbs]

Thanks for the feedback @raphaelauv ! We wrestled with the length and complexity of the blog post a lot. And decided that a lot of the details can be found in the project readme (which is probably too large now and should be split from readme and docs).

And the performance demonstrations can be found here: https://github.com/confluentinc/parallel-consumer#performance

Of course the performance numbers can be made to look faster, or they can be made to look massively faster - al depends on the use case. But I really world benchmark would be awesome, but I haven't had time to really code up something realistic except for the database test in the integration test suite, which isn't all that "real world". If you or @JorgenRingen end up doing something like that though, that would be awesome! We could probably look at doing a blog post on the confluent blog with it.

The efficiency stuff can be found in the scenarios and FAQ "Why not just run more..."

I'll take your feedback for the talk though..

[JorgenRingen]

@ismarslomic did some testing against elastic, but found elastic to be the bottleneck without being able to batch (#18)

In one of my use-cases latency of external http-endpoints (500-1000ms) is the limiting factor. The load is not extreme by any means, but even scaling a fairly low load would probably require partition numbers in the hundreds (fairly easy to calculate), which has broker and rebalance implications.
The characteristics of "pool-processing" compared to "partition-processing" is also very appealing as we don't want an "x-number-of-seconds" http-timeout to pause processing for all records on that partition.

So I agree, "efficient" is complicated and context-specific. An in-depth blog with real-world examples would be awesome :-)

[astubbs]

Perhaps, if you'd like to submit a PR for the readme to make those advantages more clear to a new user, I'd very much appreciate it :)