Random ssl handshake failed exception

[TheMidgardWatcher]

Hi, @ah- !

I'm getting ssl exceptions randomly when using ssl security protocol:
-181|2017-01-11 15:08:17.642|Test#consumer-4|FAIL| ssl://xxx:9093/bootstrap: SSL handshake failed: .\ssl\s3_both.c:406: error:1408E0F4:SSL routines:ssl3_get_message:unexpected message: : client authentication might be required (see broker log) Error: UNKNOWN_TOPIC_OR_PART

Am I doing something wrong or it's a bug?

[ah-]

Hi, it looks like this fails inside the underlying librdkafka. Could you maybe try asking at https://github.com/edenhill/librdkafka?

[TheMidgardWatcher]

Hi again! I asked @edenhill about this error, but no results... Also i've asked our other teams and they say that python client works fine with librdkafka and they didn't get any errors about ssl.

[edenhill]

Is confluent-kafka-python working with the same configuration? On the same machine?

[TheMidgardWatcher]

Actually it's other project that uses same kafka environment, same client config (except client certificates). We are using different machines and environments for our projects.

UPD: Just asked them - they are using same client certificates as we do

[edenhill]

So all config and all certs et.al are identical? (verify with md5sum or similar)
librdkafka version is also identical? (which version?)

The only things that are different are:

• dotnet instead of python ?
• windows instead of linux ?

[TheMidgardWatcher]

librdkafka - they are using some RC version of 0.9.2, but 0.9.1 also works fine for them.
config - identical (except ClientId and GroupId)
Certificates - they are using jks keystores, but they say client certificates also working as well

Yes differences only in platform and OS

[edenhill]

What are the OpenSSL versions on the respective platforms?

[TheMidgardWatcher]

python-lib - OpenSSL 1.0.1 14 Mar 2012
Kafka - OpenSSL 1.0.1t 3 May 2016

We didn't use OpenSSL - rdkafka works without it, but even if i have it installed - nothing changes.
Curently i've installed locally 1.0.2j version

[edenhill]

Let's move this issue over to librdkafka.
Please create an issue here: https://github.com/edenhill/librdkafka/issues/new

Set up your brokers with Java SSL debugging (as previously instructed on gitter) and provide relevant parts of broker logs and stderr/stdout showing:

• a connecting librdkafka-based SSL client that works
• a connecting librdkafka-based SSL client that does not work

Run both of these clients with debug=ssl (librdkafka property) and provide log/debug output.

Please correletate client and broker logs.

[TheMidgardWatcher]

Ok, i'll create and issue there when I'll get all the logs

[TheMidgardWatcher]

Hey, @edenhill just digging logs from kafka and found this:
kafka-network-thread-63-SSL-5, fatal error: 80: problem unwrapping net record
javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
kafka-network-thread-63-SSL-5, SEND TLSv1.2 ALERT: fatal, description = internal_error
kafka-network-thread-63-SSL-5, WRITE: TLSv1.2 Alert, length = 2
kafka-network-thread-63-SSL-5, called closeOutbound()
kafka-network-thread-63-SSL-5, closeOutboundInternal()
kafka-network-thread-63-SSL-5, called closeInbound()
kafka-network-thread-63-SSL-5, fatal: engine already closed. Rethrowing javax.net.ssl.SSLException: Inbound closed before receiving peer's close_notify: possible truncation attack?
kafka-network-thread-63-SSL-5, called closeOutbound()
kafka-network-thread-63-SSL-5, closeOutboundInternal()
it's only one place that looks strange and connected to ssl
Python client runs without such messages

[ah-]

Could you maybe try using the python librdkafka.dll with the C# wrapper? Just replace the file.

The one shipped with rdkafka has a statically compiled in OpenSSL, which is why your installed version makes no difference.

[TheMidgardWatcher]

Python guys that librdkafka builds into .so file not .dll so we can't use it

[TheMidgardWatcher]

@ah- When do you plan to update Internal Rdkafka-dotnet to 0.9.2 librdkafka? Maybe it will be a resolution of this issue...

[treziac]

@TheMidgardWatcher for info, you can use RdKafka prerelease to target librdkafka 0.9.2 (to check if it resolves your issue)

[TheMidgardWatcher]

@treziac - already tried ci builds of rdkafka-dotnet - no result

[ah-]

The latest ci builds already use a final 0.9.2 librdkafka shared library.

Could you try using some other 0.9.2 librdkafka.dll? I suspect the issue might be with how the bundled one got built, maybe due to OpenSSL or something else.

[TheMidgardWatcher]

Hi there, @ah-!

I've already tried almost all ci builds of rdkafka-dotnet - from 0.9.2-ci-170 to 0.9.2-ci-186.
Same result - ssl errors

[treziac]

I think he meant to retrieve (or build) librdkafka.dll and put it in the folder for execution (instead of the one coming from RdKafka.Internal which is packed in RdKafka, found in folders x64 or x86). For example, the current packaged dll does not include LZ4, it might also not support openssl correctly (didn't use it so can't confirm)

[ah-]

Hi, yeah that was the idea, all the binaries on nuget were built on the same CI server, so they might all have the same issues. A different build, especially with a different OpenSSL might change it.

[TheMidgardWatcher]

Hi again!

@treziac , @ah- I've already tried to compile librdkafka.dll but when i rplaced it i got error about loading image with incorrect format or so

[ah-]

Are you building for the same architecture? 64/32bit?

[TheMidgardWatcher]

Yes, I built both libs - 32/64 and tried them with same result.