Playground for Kafka/Confluent Docker experimentations...
ℹ️ How to run
- 🔗 Kafka Connectors
- ☁️ Confluent Cloud
- 🔄 Confluent Replicator and Mirror Maker 2
- 🔐 Environments
- Confluent Commercial
- CP-Ansible Playground
- 👾 Other Playgrounds
- 📚 Useful Resources
Quick start examples from Confluent docs but in Docker version for ease of use.
| Connector | Product Category | Latest Version (*) | Type | Release Date |  Travis |
|---|---|---|---|---|---|
 ActiveMQ Sink |
Message Queue | :confluentinc-kafka-connect-activemq-sink: | |||
| ActiveMQ Source |
Message Queue | :confluentinc-kafka-connect-activemq: | |||
 Amazon CloudWatch Logs Source |
Analytics | :confluentinc-kafka-connect-aws-cloudwatch-logs: | |||
| Amazon CloudWatch Metrics Sink |
Analytics | :confluentinc-kafka-connect-aws-cloudwatch-metrics: | |||
 Amazon DynamoDB Sink |
Database | :confluentinc-kafka-connect-aws-dynamodb: | |||
 Amazon Kinesis Source |
Message Queue | :confluentinc-kafka-connect-kinesis: | |||
 Amazon Redshift Sink |
Data Warehouse | :confluentinc-kafka-connect-aws-redshift: | |||
| Amazon Redshift Source (using JDBC) |
Data Warehouse | :confluentinc-kafka-connect-jdbc: | |||
 Amazon S3 Sink |
Datastore | :confluentinc-kafka-connect-s3: | |||
| Amazon S3 Source |
Datastore | :confluentinc-kafka-connect-s3-source: | |||
 Amazon SQS Source |
Message Queue | :confluentinc-kafka-connect-sqs: | |||
 AMPS Source |
Message Queue | :confluentinc-kafka-connect-amps: | |||
 Apache Kudu Source |
Database | :confluentinc-kafka-connect-kudu: | |||
| Apache Kudu Sink |
Database | :confluentinc-kafka-connect-kudu: | |||
 AWS Lambda Sink |
SaaS Apps | :confluentinc-kafka-connect-aws-lambda: | |||
 Azure Blob Storage Sink |
Datastore | :confluentinc-kafka-connect-azure-blob-storage: | |||
| Azure Blob Storage Source |
Datastore | :confluentinc-kafka-connect-azure-blob-storage-source: | |||
 Azure Data Lake Storage Gen1 Sink |
Datastore | :confluentinc-kafka-connect-azure-data-lake-gen1-storage: | |||
| Azure Data Lake Storage Gen2 Sink |
Datastore | :confluentinc-kafka-connect-azure-data-lake-gen2-storage: | |||
 Azure Event Hubs Source |
Message Queue | :confluentinc-kafka-connect-azure-event-hubs: | |||
 Azure Search Sink |
Analytics | :confluentinc-kafka-connect-azure-search: | |||
 Azure Service Bus Source |
Message Queue | :confluentinc-kafka-connect-azure-service-bus: | |||
 Azure SQL Data Warehouse Sink |
Data Warehouse | :confluentinc-kafka-connect-azure-sql-dw: | |||
 Cassandra Sink |
Database | :confluentinc-kafka-connect-cassandra: | |||
 Couchbase Sink |
Database | :kafka-connect-couchbase: | |||
| Couchbase Source |
Database | :kafka-connect-couchbase: | |||
 Debezium CDC Microsoft SQL Server Source |
CDC | :debezium-debezium-connector-sqlserver: | |||
 Debezium CDC MySQL Source |
CDC | :debezium-debezium-connector-mysql: | |||
 Debezium CDC PostgreSQL Source |
CDC | :debezium-debezium-connector-postgresql: | |||
 Debezium CDC MongoDB Source |
CDC | :debezium-debezium-connector-mongodb: | |||
 Data Diode Sink |
Logs | :confluentinc-kafka-connect-data-diode: | |||
| Data Diode Source |
Logs | :confluentinc-kafka-connect-data-diode: | |||
 Datadog Metrics Sink |
Analytics | :confluentinc-kafka-connect-datadog-metrics: | |||
 ElasticSearch Sink |
Analytics | :confluentinc-kafka-connect-elasticsearch: | |||
 FTPS Sink |
Datastore | :confluentinc-kafka-connect-ftps: | |||
| FTPS Source |
Datastore | :confluentinc-kafka-connect-ftps: | |||
 Gemfire Sink |
Database | :confluentinc-kafka-connect-pivotal-gemfire: | |||
 Github Source |
SaaS Apps | :confluentinc-kafka-connect-github: | |||
 Google BigQuery Sink |
Data Warehouse | :wepay-kafka-connect-bigquery: | |||
 Google Cloud Functions Sink |
SaaS Apps | :confluentinc-kafka-connect-gcp-functions: | |||
 Google Cloud Pub/Sub Source |
Message Queue | :confluentinc-kafka-connect-gcp-pubsub: | |||
 Google Cloud Spanner Sink |
Database | :confluentinc-kafka-connect-gcp-spanner: | |||
 Google Cloud Storage Sink |
Datastore | :confluentinc-kafka-connect-gcs: | |||
| Google Cloud Storage Source |
Datastore | :confluentinc-kafka-connect-gcs-source: | |||
 Google Firebase Realtime Database Sink |
Database | :confluentinc-kafka-connect-firebase: | |||
| Google Firebase Realtime Database Source |
Database | :confluentinc-kafka-connect-firebase: | |||
 HBase Sink |
Database | :confluentinc-kafka-connect-hbase: | |||
 HDFS 2 Source |
Datastore | :confluentinc-kafka-connect-hdfs2-source: | |||
| HDFS 3 Source |
Datastore | :confluentinc-kafka-connect-hdfs3-source: | |||
| HDFS 2 Sink |
Datastore | :confluentinc-kafka-connect-hdfs: | |||
| HDFS 3 Sink |
Datastore | :confluentinc-kafka-connect-hdfs3: | |||
 HTTP Sink |
SaaS Apps | :confluentinc-kafka-connect-http: | |||
 IBM MQ Sink |
Message Queue | :confluentinc-kafka-connect-ibmmq-sink: | |||
| IBM MQ Source |
Message Queue | :confluentinc-kafka-connect-ibmmq: | |||
 InfluxDB Sink |
Database | :confluentinc-kafka-connect-influxdb: | |||
| InfluxDB Source |
Database | :confluentinc-kafka-connect-influxdb: | |||
 JDBC Hive Sink |
Database | :confluentinc-kafka-connect-jdbc: | |||
| JDBC MySQL Sink |
Database | :confluentinc-kafka-connect-jdbc: | |||
 JDBC Oracle 11 Sink |
Database | :confluentinc-kafka-connect-jdbc: | |||
 JDBC Oracle 12 Sink |
Database | :confluentinc-kafka-connect-jdbc: | |||
| JDBC PostGreSQL Sink |
Database | :confluentinc-kafka-connect-jdbc: | |||
| JDBC Microsoft SQL Server Sink |
Database | :confluentinc-kafka-connect-jdbc: | |||
 JDBC Vertica Sink |
Database | :confluentinc-kafka-connect-jdbc: | |||
| JDBC MySQL Source |
Database | :confluentinc-kafka-connect-jdbc: | |||
| JDBC Oracle 11 Source |
Database | :confluentinc-kafka-connect-jdbc: | |||
| JDBC Oracle 12 Source |
Database | :confluentinc-kafka-connect-jdbc: | |||
| JDBC PostGreSQL Source |
Database | :confluentinc-kafka-connect-jdbc: | |||
| JDBC Microsoft SQL Server Source |
Database | :confluentinc-kafka-connect-jdbc: | |||
 JIRA Source |
SaaS Apps | :confluentinc-kafka-connect-jira: | |||
| JMS ActiveMQ Sink |
Message Queue | :confluentinc-kafka-connect-jms-sink: | |||
 JMS Solace Sink |
Message Queue | :confluentinc-kafka-connect-jms-sink: | |||
 JMS TIBCO EMS Sink |
Message Queue | :confluentinc-kafka-connect-jms-sink: | |||
| JMS TIBCO EMS Source |
Message Queue | :confluentinc-kafka-connect-jms: | |||
 Mapr Sink |
Datastore | :confluentinc-kafka-connect-maprdb: | |||
 Minio Sink |
Datastore | :confluentinc-kafka-connect-s3: | |||
| MongoDB Sink |
Database | :mongodb-kafka-connect-mongodb: | |||
| MongoDB Source |
Database | :mongodb-kafka-connect-mongodb: | |||
 MQTT Sink |
IoT | :confluentinc-kafka-connect-mqtt: | |||
| MQTT Source |
IoT | :confluentinc-kafka-connect-mqtt: | |||
 Neo4j Sink |
Database | :neo4j-kafka-connect-neo4j: | |||
 OmniSci Sink |
Database | :confluentinc-kafka-connect-omnisci: | |||
 PagerDuty Sink |
SaaS Apps | :confluentinc-kafka-connect-pagerduty: | |||
 Prometheus Sink |
Analytics | :confluentinc-kafka-connect-prometheus-metrics: | |||
 RabbitMQ Sink |
Message Queue | :confluentinc-kafka-connect-rabbitmq-sink: | |||
| RabbitMQ Source |
Message Queue | :confluentinc-kafka-connect-rabbitmq: | |||
 Redis Sink |
Database | :jcustenborder-kafka-connect-redis: | |||
 SalesForce Bulk API Sink |
SaaS Apps | :confluentinc-kafka-connect-salesforce-bulk-api: | |||
| SalesForce Bulk API Source |
SaaS Apps | :confluentinc-kafka-connect-salesforce-bulk-api: | |||
| SalesForce CDC Source |
SaaS Apps | :confluentinc-kafka-connect-salesforce: | |||
| SalesForce Platform Events Sink |
SaaS Apps | :confluentinc-kafka-connect-salesforce: | |||
| SalesForce Platform Events Source |
SaaS Apps | :confluentinc-kafka-connect-salesforce: | |||
| SalesForce PushTopics Source |
SaaS Apps | :confluentinc-kafka-connect-salesforce: | |||
| SalesForce SObject Sink |
SaaS Apps | :confluentinc-kafka-connect-salesforce: | |||
 ServiceNow Sink |
SaaS Apps | :confluentinc-kafka-connect-servicenow: | |||
| ServiceNow Source |
SaaS Apps | :confluentinc-kafka-connect-servicenow: | |||
 SFTP Sink |
Datastore | :confluentinc-kafka-connect-sftp: | |||
| SFTP Sink |
Datastore | :confluentinc-kafka-connect-sftp: | |||
 SNMP Trap Source |
IoT | :confluentinc-kafka-connect-snmp: | |||
 Snowflake Sink |
Data Warehouse | :snowflakeinc-snowflake-kafka-connector: | |||
| Solace Sink |
Message Queue | :confluentinc-kafka-connect-solace-sink: | |||
| Solace Source |
Message Queue | :confluentinc-kafka-connect-solace-source: | |||
 Splunk Sink |
Analytics | :splunk-kafka-connect-splunk: | |||
| Splunk Source |
Analytics | :confluentinc-kafka-connect-splunk-source: | |||
 Spool Dir Source |
Datastore | :jcustenborder-kafka-connect-spooldir: | |||
 Syslog Source |
Logs | :confluentinc-kafka-connect-syslog: | |||
| TIBCO EMS Sink |
Message Queue | :confluentinc-kafka-connect-tibco-sink: | |||
| TIBCO EMS Source |
Message Queue | :confluentinc-kafka-connect-tibco-source: | |||
| Vertica Sink |
Database | :confluentinc-kafka-connect-vertica: | |||
 Zendesk Source |
SaaS Apps | :confluentinc-kafka-connect-zendesk: |
* You can change default connector version by setting CONNECTOR_TAG environment variable before starting a test, get more details here
- How to connect your components to Confluent Cloud
- How to monitor your Confluent Cloud cluster
- How to restrict access
- etc...
ServiceNow source- ServiceNow sink
MongoDB source
Firebase
- Using cp-ansible with Confluent Cloud
- Demo using dabz/ccloudexporter in order to pull Metrics API data from Confluent Cloud cluster and export it to Prometheus (Grafana dashboard is also available)
.NET client (producer/consumer)
Go client (producer/consumer)kafka-admin Managing topics and ACLs using matt-mangia/kafka-admin
Confluent Replicator OnPrem to cloud and Cloud to Cloud examples
Multi-Cluster Schema Registry with hybrid configuration (onprem/confluent cloud)
- Confluent REST Proxy Security Plugin with Principal Propagation
- Migrate Schemas to Confluent Cloud using Confluent Replicator
Using Multi-Data-Center setup with US 🇺🇸 and EUROPE 🇪🇺 clusters.
Using Confluent Replicator as connector
- Using PLAINTEXT
- Using SASL_PLAIN
- Using Kerberos
- 👾 Using Confluent Replicator as executable
- Using PLAINTEXT
- Using SASL_PLAIN
- Using Kerberos
Using Mirror Maker 2
- Using PLAINTEXT
Single cluster:
- PLAINTEXT: no security
- SASL_PLAIN: no SSL encryption, SASL/PLAIN authentication
- SASL/SCRAM no SSL encryption, SASL/SCRAM-SHA-256 authentication
- SASL_SSL: SSL encryption, SASL/PLAIN authentication
- 2WAY_SSL: SSL encryption, SSL authentication
- Kerberos: no SSL encryption, Kerberos GSSAPI authentication
- SSL_Kerberos SSL encryption, Kerberos GSSAPI authentication
- LDAP Authorizer with SASL/PLAIN no SSL encryption, SASL/PLAIN authentication, LDAP Authorizer for ACL authorization
- RBAC with SASL/PLAIN RBAC with no SSL encryption, SASL/PLAIN authentication
Multi-Data-Center setup:
- PLAINTEXT: no security
- SASL_PLAIN: no SSL encryption, SASL/PLAIN authentication
- Kerberos: no SSL encryption, Kerberos GSSAPI authentication
- Control Center
- Tiered Storage
- Tiered storage with AWS S3
- Tiered storage with Minio (unsupported)
- Confluent Rebalancer
- JMS Client
- RBAC with SASL/PLAIN RBAC with no SSL encryption, SASL/PLAIN authentication
- Audit Logs
- Confluent REST Proxy Security Plugin with SASL_SSL and 2WAY_SSL Principal Propagation
Easily play with Confluent Platform Ansible playbooks by using Ubuntu based Docker images generated daily from this cp-ansible-playground repository
There is also a Confluent Cloud version available here
- Confluent Replicator [also with SASL_SSL and 2WAY_SSL]
- Testing Separate principals (
connector.client.config.override.policy) for Source connector (SFTP source) - Testing Separate principals (
connector.client.config.override.policy) for Source connector (SFTP sink) - How to write logs to files when using docker-compose
- Publish logs to kafka with Elastic Filebeat
Monitor Confluent Platform with Datadog
- A Kafka Story 📖: A step by step guide to use Kafka ecosystem (Kafka Connect, KSQL, Java Consumers/Producers, etc..) with Docker
- Kafka Boom Boom 💥: An attempt to break kafka
- Kafka Security playbook 🔒: demonstrates various security configurations with Docker
- MDC and single views 🌍: Multi-Data-Center setup using Confluent Replicator
- Kafka Platform Prometheus 📊: Simple demo of how to monitor Kafka Platform using Prometheus and Grafana.