How to enable SSL with rdkafka-dotnet

[mikebutler]

I have just started using this library and cannot find any documentation on how to setup SSL with rdkafka-dotnet.

[ah-]

See https://github.com/edenhill/librdkafka/wiki/Using-SSL-with-librdkafka and https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md

Basically you need to set these config options:

security.protocol=ssl

# CA certificate file for verifying the broker's certificate.
ssl.ca.location=ca-cert

# Client's certificate
ssl.certificate.location=client_?????_client.pem

# Client's key
ssl.key.location=client_?????_client.key

# Key password, if any.
ssl.key.password=abcdefgh

[mikebutler]

thank you very much, i'll give it a try.

[ah-]

Let me know how it goes, I'm interested! Enabling the debug config setting might help if it doesn't just work.

[MaximGurschi]

Are you running this on Windows or Linux?

[mikebutler]

I am running the producers / consumers on windows 10 and the kafka brokers / zookeeper are running in redhat

[MaximGurschi]

Cheers!

[rajputs6]

Hello ,
I am new at Kafka server and just found your api to work for . I have requirement to connect to kakfa over SSL , but really do not have idea . Luckly found this thread. Although i quickly got api capabilities and figured it out that the configs options needs to be set while creating config object, but confusion on values of these configuration options for example ssl.ca.location=ca-cert , can you please guide what is ca-cert . Is it some directory ? if yes then what should I place in it

Please also provide some details on
ssl.certificate.location=client_?????client.pem
ssl.key.location=client?????_client.key

I am sorry if this question is too basic . But i will be gr8ful if you can guide .

[mikebutler]

Hey,

I have played around with this and I have got the kafka over SSL working. I did not go as far as configuring topic permissions based on the client certificate but I can help get you started with getting kafka working over SSL. The end result for me ended up being one port for external access using SSL and another port for internal services along with communication between brokers as plaintext. I found that the SSL very negatively affects performance so I left the internal traffic with plaintext.

I used the following guides:
http://docs.confluent.io/2.0.0/kafka/ssl.html
http://kafka.apache.org/documentation.html#security_ssl

I created my own CA(Certificate Authority) to self sign my other certificates. Follow the guide from the first link I sent you, I basically followed that step by step.

Good Luck!

[rajputs6]

Thank you for the detailed response. the step by step guide is perfect. Last think . Is there special configs / step which i need to take care while connecting to Kafka over SSL using rdkafka-dotnet api. Can you please provide information with respect to rdkafka-dotnet api usage with SSL. What extra code is required ?

In my current scenraio the staging environment is already setup without SSL and rdkafka-dotnet api works perfect. But the production is on SSL .

Thank you in advance

[mikebutler]

Add the following to the config object passed into the producer / eventconsumer constructors.
config["security.protocol"] = "ssl";
config["ssl.ca.location"] = @"[ca-cert path]";
config["ssl.certificate.location"] = @"[cert-signed path]";
config["ssl.key.password"] = "[password from the step by step guide]";

[Disamis]

According to this guide we should use following client config options:
security.protocol=SSL
ssl.truststore.location=/var/private/ssl/kafka.client.truststore.jks
ssl.truststore.password=test1234

But when I use it in C# I have a error:
clientConfig["security.protocol"] = "SSL";
clientConfig["ssl.truststore.location"] = "c:\kafka\kafka_2.11-0.10.1.0\kafka.client.truststore.jks";
clientConfig["ssl.truststore.password"] = "[pass]";

Error: No such configuration property: "ssl.truststore.location"

My kafka server is configured with truststore. I have only truststore.jks file. When I use this config from CMD it works fine, but from C# it leads to error. How I can connect to kafka server, when I have only truststore.jks file?
Thank you!

[edenhill]

rdkafka-dotnet, which is based on librdkafka, does not use Java key stores.
See this wiki for details:
https://github.com/edenhill/librdkafka/wiki/Using-SSL-with-librdkafka

[mikebutler]

Disamis,
Please set these settings:
config["security.protocol"] = "ssl";
config["ssl.ca.location"] = @"[ca-cert path]";
config["ssl.certificate.location"] = @"[cert-signed path]";
config["ssl.key.password"] = "[password from the step by step guide]";

The step by step would have generated a ca-cert file and a cert-signed file. Use those files with the above settings and it should work.

[Disamis]

Thank you all. It works with ca-cert and cert-signed files.

[mikebutler]

Good to hear. If you take it any further and setup different client certs with access to different topics please post your results here.