MinIO and Koperator

How MinIO integrates with Koperator in k8s

Documentation based on:

• https://kafka.apache.org/quickstart
• install-kafka-operator
• https://github.com/banzaicloud/koperator
• https://banzaicloud.com/docs/supertubes/kafka-operator/test/
• https://docs.min.io/docs/minio-bucket-notification-guide.html

Using minikube

1. Start cluster, making sure enough space is allowed (minimum 6 vCPU and 10 GB RAM):

minikube stop
minikube delete
minikube start --memory=10240

2. Install cert-manager:

kubectl create -f https://github.com/jetstack/cert-manager/releases/download/v1.6.2/cert-manager.yaml

3. Install Zookeeper:

helm repo add pravega https://charts.pravega.io
helm repo update
helm install zookeeper-operator --namespace=zookeeper --create-namespace pravega/zookeeper-operator
kubectl create --namespace zookeeper -f - <<EOF
apiVersion: zookeeper.pravega.io/v1beta1
kind: ZookeeperCluster
metadata:
    name: zookeeper
    namespace: zookeeper
spec:
    replicas: 1
EOF

4. Install Koperator with Helm:

kubectl create --validate=false -f https://github.com/banzaicloud/koperator/releases/download/v0.21.0/kafka-operator.crds.yaml
helm repo add banzaicloud-stable https://kubernetes-charts.banzaicloud.com/
helm repo update
helm install kafka-operator --namespace=kafka --create-namespace banzaicloud-stable/kafka-operator
kubectl create -n kafka -f https://raw.githubusercontent.com/banzaicloud/koperator/master/config/samples/simplekafkacluster.yaml

5. Create topic (wait until kafka pods are ready):

cat << EOF | kubectl apply -n kafka -f -
apiVersion: kafka.banzaicloud.io/v1alpha1
kind: KafkaTopic
metadata:
    name: my-topic
spec:
    clusterRef:
        name: kafka
    name: my-topic
    partitions: 1
    replicationFactor: 1
    config:
        "retention.ms": "604800000"
        "cleanup.policy": "delete"
EOF

6. Produce messages in window 1:

kubectl -n kafka run kafka-producer -it --image=banzaicloud/kafka:2.13-2.4.0 --rm=true --restart=Never -- /opt/kafka/bin/kafka-console-producer.sh --broker-list kafka-headless:29092 --topic my-topic

You should see:

$ kubectl -n kafka run kafka-producer -it --image=banzaicloud/kafka:2.13-2.4.0 --rm=true --restart=Never -- /opt/kafka/bin/kafka-console-producer.sh --broker-list kafka-headless:29092 --topic my-topic
If you don't see a command prompt, try pressing enter.
>event1
>event2
>event3
>

7. Consume messages in window 2:

kubectl -n kafka run kafka-consumer -it --image=banzaicloud/kafka:2.13-2.4.0 --rm=true --restart=Never -- /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server kafka-headless:29092 --topic my-topic --from-beginning

You should see:

$ kubectl -n kafka run kafka-consumer -it --image=banzaicloud/kafka:2.13-2.4.0 --rm=true --restart=Never -- /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server kafka-headless:29092 --topic my-topic --from-beginning
If you don't see a command prompt, try pressing enter.

event1
event2
event3

8. Install MinIO Operator and MinIO Tenant:

kubectl apply -k ~/operator/resources
kubectl apply -k ~/operator/examples/kustomization/tenant-lite

9. Forward MinIO port (Wait for MinIO pods to be ready):

kubectl port-forward storage-lite-pool-0-0 -n tenant-lite 9000

10. Add MinIO alias:

mc alias set myminio https://localhost:9000 minio minio123 --insecure

You should see:

$ mc alias set myminio https://localhost:9000 minio minio123 --insecure
Added `myminio` successfully.

11. Add Kafka endpoint to MinIO

mc admin config set myminio notify_kafka:1 tls_skip_verify="off"  \
queue_dir="" queue_limit="0" sasl="off" sasl_password="" sasl_username="" \
tls_client_auth="0" tls="off" client_tls_cert="" client_tls_key="" \
brokers="172.17.0.11:29093" topic="my-topic" version="" --insecure

You should see:

mc admin config set myminio notify_kafka:1 tls_skip_verify="off"  queue_dir="" queue_limit="0" sasl="off" sasl_password="" sasl_username="" tls_client_auth="0" tls="off" client_tls_cert="" client_tls_key="" brokers="172.17.0.11:29093" topic="my-topic" version="" --insecure
Successfully applied new settings.
Please restart your server 'mc admin service restart myminio'.

12. Restart MinIO:

mc admin service restart myminio --insecure

13. Enable Kafka bucket notification using MinIO client

mc mb myminio/images --insecure
mc event add  myminio/images arn:minio:sqs::1:kafka --suffix .jpg --insecure
mc event list myminio/images --insecure

You should see:

$ mc mb myminio/images --insecure
Bucket created successfully `myminio/images`.
$ mc event add  myminio/images arn:minio:sqs::1:kafka --suffix .jpg --insecure
Successfully added arn:minio:sqs::1:kafka
$ mc event list myminio/images --insecure
arn:minio:sqs::1:kafka   s3:ObjectCreated:*,s3:ObjectRemoved:*,s3:ObjectAccessed:*   Filter: suffix=".jpg"

14. Upload/copy image to folder created

mc cp rose.jpg myminio/images --insecure

15. As a result, the consumer from step 7 above will print the event:

{
  "EventName": "s3:ObjectCreated:Put",
  "Key": "images/rose.jpg",
  "Records": [
    {
      "eventVersion": "2.0",
      "eventSource": "minio:s3",
      "awsRegion": "",
      "eventTime": "2022-04-05T01:43:53.018Z",
      "eventName": "s3:ObjectCreated:Put",
      "userIdentity": {
        "principalId": "minio"
      },
      "requestParameters": {
        "principalId": "minio",
        "region": "",
        "sourceIPAddress": "127.0.0.1"
      },
      "responseElements": {
        "content-length": "0",
        "x-amz-request-id": "16E2DCA93D259AE4",
        "x-minio-deployment-id": "968b47d7-4857-4cf0-8906-58ce4716e1e6",
        "x-minio-origin-endpoint": "https://minio.tenant-lite.svc.cluster.local"
      },
      "s3": {
        "s3SchemaVersion": "1.0",
        "configurationId": "Config",
        "bucket": {
          "name": "images",
          "ownerIdentity": {
            "principalId": "minio"
          },
          "arn": "arn:aws:s3:::images"
        },
        "object": {
          "key": "rose.jpg",
          "size": 165352,
          "eTag": "c8e032a8b653aebb6b6141d50b5f3cd3",
          "contentType": "image/jpeg",
          "userMetadata": {
            "content-type": "image/jpeg"
          },
          "sequencer": "16E2DCA93DFB0DCC"
        }
      },
      "source": {
        "host": "127.0.0.1",
        "port": "",
        "userAgent": "MinIO (darwin; arm64) minio-go/v7.0.23 mc/RELEASE.2022-02-23T03-15-59Z"
      }
    }
  ]
}

Using OpenShift