How to set the "camel.sink.endpoint.sslContextParameters" in the CamelHttpsSinkConnector.properties

[KevinMengSF]

hi,

I am trying to use the CamelHttpsSinkConnector to fulfill that sync the Kafka events to the remote Endpoint.

The remote endpoint URL accessing must by MTLS need to set keystore information.

According to below example and doc, I cannot understand how to set the "camel.sink.endpoint.sslContextParameters" in the CamelHttpsSinkConnector.properties
Could you provide one example for that?

https://github.com/apache/camel-kafka-connector/blob/camel-kafka-connector-4.0.x/connectors/camel-https-kafka-connector/src/main/docs/examples/CamelHttpsSinkConnector.properties

https://camel.apache.org/camel-kafka-connector/next/reference/connectors/camel-https-kafka-sink-connector.html

Best regards

[oscerd]

What version of the connector are you using?

[KevinMengSF]

@oscerd Hello, sorry for replying lately, the version destais as: implementation group: 'org.apache.camel.kafkaconnector', name: 'camel-https-kafka-connector', version: '4.0.3'.

[oscerd]

The implementation is based on the following Kamelet https://github.com/apache/camel-kamelets/blob/v4.0.3/kamelets/http-secured-sink.kamelet.yaml and SSLContextParameters is not exposed.

[KevinMengSF]

@oscerd Thank you for your replying, our scenario is as below:

1. Kafka events will sink to multiple Endpoints URLs by camel-https-kafka-connector.
2. Different endpoint URL has different auth Methods some of use "keystore information" for MTLS, some of use "Basic" or "oauth2".

If SSLContextParameters is not exposed, how can we handle MTLS scenario for setting "keystore information"? Do you have any suggestion for that?

[oscerd]

We need to create a new Kameelt for that and generating a new connector for that purpose. This requires time.

[KevinMengSF]

@oscerd hope can release new connector for SSLContextParameters setting asap.

Currently we cannot directly do the configuration for SSLContextParameters in the file:
https://github.com/apache/camel-kafka-connector/blob/camel-kafka-connector-4.0.x/connectors/camel-https-kafka-connector/src/main/docs/examples/CamelHttpsSinkConnector.properties

or

https://github.com/apache/camel-kamelets/blob/v4.0.3/kamelets/http-secured-sink.kamelet.yaml

Whether only can create a project like Springboot Project and create bean name like "mySSLContextParameters" of SSLContextParameters according to spring configuration:
"
<camel:sslContextParameters
id="mySSLContextParameters">
<camel:keyManagers
keyPassword="keyPassword">
<camel:keyStore
resource="/users/home/server/keystore.jks"
password="keystorePassword"/>
</camel:keyManagers>
</camel:sslContextParameters>
"

And reference bean mySSLContextParameters by configuration like
" "

As this document explanation:
https://camel.apache.org/components/4.4.x/http-component.html#_setting_up_ssl_for_http_client

I am fresher for Camel Connector, my understanding is correct or not?

[oscerd]

You cannot use the approach with bean and SB project in the Kafka connect context

[oscerd]

I don't know the ETA for this.

[KevinMengSF]

@oscerd One more question, regarding below configuration file
https://github.com/apache/camel-kamelets/blob/v4.0.3/kamelets/http-secured-sink.kamelet.yaml
1. How it integrate Kafka?

2.Does it support Kafka Connect "Single Message Transform" configuration like below? if it supports could you share some samples for configuration?

"""
transforms=tenantSelection,insertHeaders

transforms.tenantSelection.type=com.services.kafka.connect.transforms.FilterByTenantId
transforms.tenantSelection.include=XXXXXX

transforms.insertHeaders.type=org.apache.kafka.connect.transforms.InsertHeader
transforms.insertHeaders.header=CamelHeader.Content-Type
transforms.insertHeaders.value.literal=application/cloudevents+json

"""

Thanks.

[oscerd]

@oscerd One more question, regarding below configuration file https://github.com/apache/camel-kamelets/blob/v4.0.3/kamelets/http-secured-sink.kamelet.yaml

Does it support Kafka Connect "Single Message Transform" configuration like below? if it supports could you share some samples for configuration?

""" transforms=tenantSelection,insertHeaders

transforms.tenantSelection.type=com.services.kafka.connect.transforms.FilterByTenantId transforms.tenantSelection.include=XXXXXX

transforms.insertHeaders.type=org.apache.kafka.connect.transforms.InsertHeader transforms.insertHeaders.header=CamelHeader.Content-Type transforms.insertHeaders.value.literal=application/cloudevents+json

"""

Thanks.

No, Kamelets are building blocks used to generate the Kafka connectors in this repository, but they are not Kafka connect oriented. If you want to use SMT, you have to extend the connector and add your own SMT.

[KevinMengSF]

@oscerd One more question, regarding below configuration file https://github.com/apache/camel-kamelets/blob/v4.0.3/kamelets/http-secured-sink.kamelet.yaml
Does it support Kafka Connect "Single Message Transform" configuration like below? if it supports could you share some samples for configuration?
""" transforms=tenantSelection,insertHeaders
transforms.tenantSelection.type=com.services.kafka.connect.transforms.FilterByTenantId transforms.tenantSelection.include=XXXXXX
transforms.insertHeaders.type=org.apache.kafka.connect.transforms.InsertHeader transforms.insertHeaders.header=CamelHeader.Content-Type transforms.insertHeaders.value.literal=application/cloudevents+json
"""
Thanks.

No, Kamelets are building blocks used to generate the Kafka connectors in this repository, but they are not Kafka connect oriented. If you want to use SMT, you have to extend the connector and add your own SMT.

How does http-secured-sink.kamelet.yaml integrate with Kafka? I cannot find any config related with Kafka? Does it need combine with https://github.com/apache/camel-kamelets/blob/v4.0.3/kamelets/kafka-source.kamelet.yaml ? Could you show some demo for that?

[oscerd]

Not http specific, but you can have a look at https://github.com/apache/camel-kamelets-examples/tree/main/jbang/kafka-batch-s3

[KevinMengSF]

@oscerd Regarding Kafka sink connector of camel-https-kafka-connector doc reference :https://camel.apache.org/camel-kafka-connector/4.0.x/reference/connectors/camel-https-kafka-sink-connector.html

How does it support oauth2 relevant configuration items something like below http component? Not sure whether it is similar with SSLContextParameters also does not expose to configuration file.
https://camel.apache.org/components/4.4.x/http-component.html#_oauth2_support.

[oscerd]

It doesn't expose those parameters. We need a specialized kamelet for that purpose and we need to support camel 4.4.x

[KevinMengSF]

@oscerd There is another issue regarding camel-https-kafka-connector,
we know the connector forwards the message (record) value to the HTTP API.

Not sure we can add parameters to have the connector construct a unique HTTP API URL like containing the Kafka record key , topic name and others value that refer to fields from the Kafka record.
For example, we enter http://test:9200/api/messages/${topic}/${key} ?id={id} to have the HTTP API URL contain the topic name, record key and id like "http://test:9200/api/messages/test_topic/key_value?id=3c0aa2b2-ffb8-4855-863e-6edcc8985f67"

If currently we do not support that, whether have plan to implement it?

Thanks

[oscerd]

For doing that we need to use toD in the Kamelet instead of to and you'll need to use Camel headers for the purpose. As of today we don't have any plan to support that, first we need to switch to Camel 4.4.x

[KevinMengSF]

@oscerd

One more question regarding CamelHttpsSinkConnector configuration.
Could you please help confirm the property "camel.component.https.cookieManagementDisabled" in the file of https://camel.apache.org/camel-kafka-connector/next/reference/connectors/camel-https-kafka-sink-connector.html, whether it is exposed or not?

I do not know how to configure it in the properties file.

Version destais as: implementation group: 'org.apache.camel.kafkaconnector', name: 'camel-https-kafka-connector', version: '4.0.3'.

Thanks

[oscerd]

It's not exposed.

[KevinMengSF]

@oscerd Now, shall we have some methods or approches to disableCookieManagement?
There is an use case need to disable cookie in the CamelHttpsSinkConnector for HttpClient Object, like below code:

CloseableHttpClient httpClient = HttpClients.custom()
.disableCookieManagement()
.build();

[KevinMengSF]

It's not exposed.

@oscerd For the Camel 4.4.x, shall we can expose the "camel.component.https.cookieManagementDisabled" to control the Apache HttpClient can disable the cookie management?

We have the use case that the events are belong to multi-tenants, during events sync to Remote Endpoint, it will cause the issue something like the tenantA’s cookie will send to tenantB’s and lead security issue.
So to avoid this security issue, we want to disableCookieManagement for Apache HttpClient.

Thanks

[oscerd]

I created this: apache/camel-kamelets#2042

The target it's 4.7.0, which won't be an LTS, @valdar is working on releasing 4.4.x for Camel-Kafka-connector and the next supported LTS will 4.8.0. So you'll have to wait.

[KevinMengSF]

I created this: apache/camel-kamelets#2042

The target it's 4.7.0, which won't be an LTS, @valdar is working on releasing 4.4.x for Camel-Kafka-connector and the next supported LTS will 4.8.0. So you'll have to wait.

@oscerd Cloud you provide the available timeline regarding the version 4.4.x and 4.8.0? we can do the further plan.
Thanks.

[oscerd]

In 4.4.x you don't have the fix. For 4.8 the release should be after summer maybe Sept/Oct. After that we need to release the kamelets and after camel Kafka connector. It takes a while

[KevinMengSF]

@oscerd Whether we not only expose "sslContextParameters","camel.component.https.cookieManagementDisabled", and add another fields as many as we can, something like but not limit "camel.sink.endpoint.authMethodPriority", "camel.sink.endpoint.authPassword", "camel.sink.endpoint.authUsername", "camel.component.https.connectionRequestTimeout","camel.component.https.connectTimeout","camel.component.https.responseTimeout","camel.component.https.soTimeout"

[KevinMengSF]

It doesn't expose those parameters. We need a specialized kamelet for that purpose and we need to support camel 4.4.x

Hi @oscerd and @valdar,

Just checked the version 4.4.2 already released, could help confirm that "sslContextParameters" and "oauth2 related parameters" whether exposed or not?

Thanks a lot.

[oscerd]

As already said:

In 4.4.x you don't have the fix. For 4.8 the release should be after summer maybe Sept/Oct. After that we need to release the kamelets and after camel Kafka connector. It takes a while

It will be in the next LTS so 4.8.x