Update aiokafka library to support higher client versions > 3

[Ben621]

The current implementation of the aiokafka library within the Confluent platform only supports client versions under 3.0 for Python. There is an error reported in aiokafka/util.py that restricts compatibility.

Additionally, the Confluent platform's version interoperability matrix highlights this limitation.

Confluent Platform	Apache Kafka®	Release Date	Standard End of Support	Platinum End of Support
7.6.x	3.6.x	February 9, 2024	February 9, 2026	February 9, 2027
7.5.x	3.5.x	August 25, 2023	August 25, 2025	August 25, 2026
7.4.x	3.4.x	May 3, 2023	May 3, 2025	May 3, 2026
7.3.x	3.3.x	November 4, 2022	November 4, 2024	November 4, 2025
7.2.x	3.2.x	July 6, 2022	July 6, 2024	July 6, 2025
7.1.x	3.1.x	April 5, 2022	April 5, 2024	April 5, 2025
7.0.x	3.0.x	October 27, 2021	October 27, 2023	October 27, 2024
6.2.x	2.8.x	June 8, 2021	June 8, 2023	June 8, 2024
6.1.x	2.7.x	February 9, 2021	February 9, 2023	February 9, 2024

We request extending compatibility to include higher client versions (e.g., 3.0 or above). This would allow users to leverage the latest features and bug fixes offered by newer aiokafka client releases.

[vmaurin]

Hi @Ben621

My company is using aiokafka against AWS MSK 3.6.0 without issue. I think the code you highlight is just happening when you force the api_version to something different to "auto"

[Ben621]

Hey @vmaurin
Can please try to print the api_version
In my case we are using 3.4.x,
But I'm unable to connect to get the api_version
But suddenly that connect, I don't know why !!
After. Investigation that work when I set api_version to 2.8.1
Maybe are you familiar with this issue ?

Also I have checked in the GitHub action they are testing client "0.9.0.1" To "2.8.1"

[vmaurin]

From the codebase, it sounds parse_kafka_version you pointed out is only use when initializing a producer or a consumer with an api_version. If you let it unset like we do (auto), this method is never called. Then when the client (aiokafka) is connecting to the broker (3.6.0), they will negotiate API version for each individual methods part of the kafka protocol.

I guess then, this parameter is to force a specific version of the protocol, but it should not be used, as Kafka protocol is build for backward and forward compatibility (even today version of the broker probably still support clients in 0.11). So my advice is to let this parameter unset

[Ben621]

Hey @vmaurin
can you please check what is you api_version when you set it auto ?

In my case after before they upgrade from kafka  2.7 to kafka 3.5 i was able to connect with auto
but after i got this error

Unable to request metadata from "kafkaserver:9093":

suddenly that work once after waiting 2 minutes for connecting and the api_version was 2.6.0

So, when i set api_version= "2.6.0" that connect directly, otherwise got error, or sometime i'm able to connect without knowing why it's happen !!!

@vmaurin BTW, according to the code you can't get broker version higher than 2.6.0

[ods]

Hi @Ben621, thank you for highlighting the problem. We need to add newer versions for the broker to the CI matrix. This will probably require reworking how we build the Docker images for them. It would be nice if somebody volunteers to do this part.

[ods]

There is an error reported in aiokafka/util.py that restricts compatibility.

Could you please provide a complete report with the traceback?

@vmaurin BTW, according to the code you can't get broker version higher than 2.6.0

The method you're referring to looks weird to me. In my opinion, we should rather drop support for brokers <0.10.0 and rely solely on what ApiVersions returns.

[wbarnha]

Hi @Ben621, thank you for highlighting the problem. We need to add newer versions for the broker to the CI matrix. This will probably require reworking how we build the Docker images for them. It would be nice if somebody volunteers to do this part.

I've been meaning to do this in https://github.com/wbarnha/kafka-python-ng for newer Kafka broker versions. If I get a working solution, I'll open a PR here as well.