Fully automated Confluent Platform Demo which deploys Kafka in multiple namespaces and uses the TF Controller with Confluent Terraform Provider to provision Confluent Cloud resources. Acompanying slides can be found here The Demo will cover:
- Minikube overview
- Deploy GitOps toolkit
- How to Install the Confluent Operator (CFK)
- Connect GitOps Source Controller to GitHub repo
- Deploy production kafka cluster via Flux
- Deploy sandbox kafka cluster
- How to Deploy Terraform Controller
- Run Terraform to provision Confluent Cloud
Many teams are already using GitOps for application deployments and now we’ll show you how you can take GitOps even further by bootstrapping whole Confluent Platform include Confluent Cloud resources straight from a Git repository. In this demo we are using Flux, Kustomize and Helm to install and manage the Confluent Operator for Kubernetes
We will configure Flux to install, deploy and config the Confluent Platform using their HelmRepository and HelmRelease custom resources. With the Confluent Operator monitoring all namespaces in the cluster, we will then allow Flux to create two clusters, in separate namespaces. Production and Sandbox, each having a slightly different configuration and number of brokers to demonstrate the ability to easily override the base configuration.
Everyone can now easily spin up their clusters in 3 simple steps: fork, clone, and run
- Fork: fork this Git repository so that you can work on it from your GitHub user account. Here’s the base repository https://github.com/osodevops/confluent-gitops-demo you can fork from.
- Clone: do git clone to download your forked repository onto your machine.
- Run: follow the instructions below to run and start your Kafka clusters.
The Toolkit components consist of Source Controller, Kustomize Controller, Helm Controller, Notification Controller and the Image Automation Controllers.
├── flux-system
│ ├── gotk-components.yaml
│ ├── gotk-sync.yaml
│ └── kustomization.yaml
├── kustomize
│ ├── base
│ │ ├── confluent
│ │ │ ├── connect
│ │ │ ├── control-centre
│ │ │ ├── kafka.yaml
│ │ │ ├── ksqldb
│ │ │ ├── kustomization.yaml
│ │ │ ├── schmea-registry
│ │ │ ├── secrets
│ │ │ └── zookeeper.yaml
│ │ └── kustomization.yaml
│ ├── environments
│ │ ├── kustomization.yaml
│ │ ├── production
│ │ └── sandbox
│ ├── operator
│ ├── terraform
│ └── tf-controller
Fork this repository into your own GitHub account.
We will be using MiniKube, but any remote cluster will work. Create the cluster using:
- Run
minikube start --cpus=6 --memory=20019 --kubernetes-version=v1.21.0
Toolkit CRDs and components are deployed using:
- Navigate to
./flux-system - Run
kubectl apply -f gotk-components.yaml
Navigate to Confluent Cloud API page and generate a new set of API keys. Copy these and head over to your terminal to run:
- `export CONFLUENT_API_KEY=REPLACE_ME && export CONFLUENT_API_SECRET=REPLACE_ME
- run ./scripts/confluent-keys.sh This will create a secret containing your keys which the Flux system can access when running Terraform via the TF Controller.
- Navigate to
./flux-system - run
kubectl apply -f gotk-sync.yamlThis next step will tell Flux what repository to monitor, and, within that repository, what kustomization files to start with. The first Kustomize resource that Flux will look for to is located at./kustomize/operator. This will install the confluent-for-kubernetes Helm chart which will monitor all namespaces for Confluent CRDs.
After a successful health check of the operator (which will run as a pod), Flux will then proceed to deploy the following Kustomizations:- Production environment located at
./kustomize/environments/production - Sandbox environment located at
./kustomize/environments/sandbox - TF Controller for reconciling Terraform resources in the GitOps way located at
./kustomize/environments/tf-controller - Auto Apply Terraform using the variables located at
./kustomize/environments/terraformYou should see the following output:
gitrepository.source.toolkit.fluxcd.io/flux-system created kustomization.kustomize.toolkit.fluxcd.io/confluent-infra created kustomization.kustomize.toolkit.fluxcd.io/production created kustomization.kustomize.toolkit.fluxcd.io/sandbox created kustomization.kustomize.toolkit.fluxcd.io/terraform-controller created kustomization.kustomize.toolkit.fluxcd.io/confluent-cloud created
- Production environment located at
Now that we have flux monitoring the forked Git repository, let's demonstrate the GitOps behaviour! If everything has deployed successfully, you should see a healthy confluent stack looking like this:
│ NAMESPACE NAME PF READY RESTARTS STATUS IP NODE AGE │
│ confluent confluent-operator-global-5fb8c9d5f8-82tqm ● 1/1 0 Running 172.17.0.11 minikube 6m39s │
│ flux-system helm-controller-544df4db67-5ghcd ● 1/1 0 Running 172.17.0.6 minikube 7m43s │
│ flux-system image-automation-controller-675bff6646-m8s5x ● 1/1 0 Running 172.17.0.4 minikube 7m43s │
│ flux-system image-reflector-controller-6bcddcfb44-dh4bv ● 1/1 0 Running 172.17.0.8 minikube 7m43s │
│ flux-system kustomize-controller-5b5bb9d49b-mh76j ● 1/1 0 Running 172.17.0.5 minikube 7m43s │
│ flux-system notification-controller-b55cbcf9c-jb8rh ● 1/1 0 Running 172.17.0.7 minikube 7m43s │
│ flux-system source-controller-6b6c7bc4bb-n29zd ● 1/1 0 Running 172.17.0.9 minikube 7m43s │
│ flux-system tf-controller-f65f5f4f5-vnrbl ● 1/1 0 Running 172.17.0.10 minikube 6m42s │
│ production connect-0 ● 1/1 0 Running 172.17.0.17 minikube 5m48s │
│ production controlcenter-0 ● 1/1 0 Running 172.17.0.13 minikube 5m49s │
│ production kafka-0 ● 1/1 0 Running 172.17.0.15 minikube 5m47s │
│ production kafka-1 ● 1/1 0 Running 172.17.0.22 minikube 5m47s │
│ production kafka-2 ● 1/1 0 Running 172.17.0.21 minikube 5m47s │
│ production schemaregistry-0 ● 1/1 0 Running 172.17.0.16 minikube 5m48s │
│ production zookeeper-0 ● 1/1 0 Running 172.17.0.20 minikube 5m47s │
│ production zookeeper-1 ● 1/1 0 Running 172.17.0.19 minikube 5m47s │
│ production zookeeper-2 ● 1/1 0 Running 172.17.0.18 minikube 5m47s │
│ sandbox kafka-0 ● 1/1 0 Running 172.17.0.12 minikube 5m49s │
│ sandbox zookeeper-0 ● 1/1 0 Running 172.17.0.14 minikube 5m48s │Check out these related projects.
- Terraform GitOps Example - A Terraform GitOps workflow showcasing how to use the TF-Controller to run your Terraform via GitOps
- Kafka GitOps Example - A Kafka / Confluent GitOps workflow example for multi-env deployments with Flux, Kustomize, Helm and Confluent Operator
- Confluent Platform on Azure - Terraform Module for deploying best practice HA Confluent Platform on Azure
File a GitHub issue, send us an email or tweet us.
Copyright © 2017-2022 OSO | See LICENCE for full details.
We at OSO help teams to adopt emerging technologies and solutions to boost their competitiveness, operational excellence and introduce meaningful innovations that drive real business growth. Our developer-first culture, combined with our cross-industry experience and battle-tested delivery methods allow us to implement the most impactful solutions for your business.
Looking for support applying emerging technologies in your business? We’d love to hear from you, get in touch by email
Start adopting new technologies by checking out our other projects, follow us on twitter, join our team of leaders and challengers, or contact us to find the right technology to support your business.