-
Notifications
You must be signed in to change notification settings - Fork 0
Home
This is a fork of the Kafka Connect Sink Connector for S3. We can use this forked repository to make required tweaks to the S3 connector, and hopefully, submit those tweaks back to the original repo to be included in a future release.
Our customizations have been added to the following branch(es) of this forked repo to support local builds:
| Branch | Description |
|---|---|
snagajob/5.3.1.x |
Additions to the 5.3.1 release of the connector plugin |
To track the remote Confluent repository:
git remote add confluent [email protected]:confluentinc/kafka-connect-storage-cloud.git
To refresh the master branch of this fork with updates from the remote repository:
git fetch confluent
git checkout master
git merge confluent/master
git push origin master
To build this project you will need to install:
To build the connector package:
mvn clean package
To deploy the connector package to MyGet:
mvn deploy:deploy-file -DgroupId=confluentinc -DartifactId=kafka-connect-s3 -Dversion=<version> -Dpackaging=zip -Dfile=kafka-connect-s3/target/components/packages/confluentinc-kafka-connect-s3-5.3.1.zip -Durl=https://www.myget.org/F/snagajob/auth/<token>/maven
Where <version> is the artifact version you want to deploy (see history below), and <token> is a MyGet token that has access to our Snagajob feed on MyGet.
Description of builds of this repository that have been deployed to MyGet.
| Version | Description |
|---|---|
| 5.3.1.1 | Added NotificationService (more details below) |
We have added the concept of a NotificationService that publishes a notification after each file is uploaded to S3, along with a KafkaNotificationService implementation that publishes those notifications to a Kafka topic.
This new service is configured using the following connector properties:
notification.kafka.enabled=true
notification.kafka.broker=localhost:9092
notification.kafka.topic=s3-notification-topic
notification.kafka.schema.registry=http://localhost:8081
notification.kafka.sasl.enabled=true (optional)
notification.kafka.sasl.username=sasl-username (optional)
notification.kafka.sasl.password=sasl-password (optional)