Skip to content

MinIO and Koperator

Jump to bottom
Cesar Celis Hernandez edited this page Apr 7, 2022 · 24 revisions

How MinIO integrates with Koperator in k8s

Documentation based on:

Using minikube

  1. Start cluster, making sure enough space is allowed (minimum 6 vCPU and 10 GB RAM):
Screen Shot 2022-04-04 at 8 03 27 PM 
minikube stop
minikube delete
minikube start --memory=10240
  1. Install cert-manager:
kubectl create -f https://github.com/jetstack/cert-manager/releases/download/v1.6.2/cert-manager.yaml
  1. Install Zookeeper:
helm repo add pravega https://charts.pravega.io
helm repo update
helm install zookeeper-operator --namespace=zookeeper --create-namespace pravega/zookeeper-operator
kubectl create --namespace zookeeper -f - <<EOF
apiVersion: zookeeper.pravega.io/v1beta1
kind: ZookeeperCluster
metadata:
    name: zookeeper
    namespace: zookeeper
spec:
    replicas: 1
EOF
  1. Install Koperator with Helm:
kubectl create --validate=false -f https://github.com/banzaicloud/koperator/releases/download/v0.21.0/kafka-operator.crds.yaml
helm repo add banzaicloud-stable https://kubernetes-charts.banzaicloud.com/
helm repo update
helm install kafka-operator --namespace=kafka --create-namespace banzaicloud-stable/kafka-operator
kubectl create -n kafka -f https://raw.githubusercontent.com/banzaicloud/koperator/master/config/samples/simplekafkacluster.yaml
  1. Create topic (wait until kafka pods are ready):
cat << EOF | kubectl apply -n kafka -f -
apiVersion: kafka.banzaicloud.io/v1alpha1
kind: KafkaTopic
metadata:
    name: my-topic
spec:
    clusterRef:
        name: kafka
    name: my-topic
    partitions: 1
    replicationFactor: 1
    config:
        "retention.ms": "604800000"
        "cleanup.policy": "delete"
EOF
  1. Produce messages in window 1:
kubectl -n kafka run kafka-producer -it --image=banzaicloud/kafka:2.13-2.4.0 --rm=true --restart=Never -- /opt/kafka/bin/kafka-console-producer.sh --broker-list kafka-headless:29092 --topic my-topic

You should see:

$ kubectl -n kafka run kafka-producer -it --image=banzaicloud/kafka:2.13-2.4.0 --rm=true --restart=Never -- /opt/kafka/bin/kafka-console-producer.sh --broker-list kafka-headless:29092 --topic my-topic
If you don't see a command prompt, try pressing enter.
>event1
>event2
>event3
>
  1. Consume messages in window 2:
kubectl -n kafka run kafka-consumer -it --image=banzaicloud/kafka:2.13-2.4.0 --rm=true --restart=Never -- /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server kafka-headless:29092 --topic my-topic --from-beginning

You should see:

$ kubectl -n kafka run kafka-consumer -it --image=banzaicloud/kafka:2.13-2.4.0 --rm=true --restart=Never -- /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server kafka-headless:29092 --topic my-topic --from-beginning
If you don't see a command prompt, try pressing enter.

event1
event2
event3
  1. Install MinIO Operator and MinIO Tenant:
kubectl apply -k ~/operator/resources
kubectl apply -k ~/operator/examples/kustomization/tenant-lite
  1. Forward MinIO port (Wait for MinIO pods to be ready):
kubectl port-forward storage-lite-pool-0-0 -n tenant-lite 9000
  1. Add MinIO alias:
mc alias set myminio https://localhost:9000 minio minio123 --insecure

You should see:

$ mc alias set myminio https://localhost:9000 minio minio123 --insecure
Added `myminio` successfully.
  1. Add Kafka endpoint to MinIO 
mc admin config set myminio notify_kafka:1 tls_skip_verify="off"  \
queue_dir="" queue_limit="0" sasl="off" sasl_password="" sasl_username="" \
tls_client_auth="0" tls="off" client_tls_cert="" client_tls_key="" \
brokers="172.17.0.11:29093" topic="my-topic" version="" --insecure

You should see:

mc admin config set myminio notify_kafka:1 tls_skip_verify="off"  queue_dir="" queue_limit="0" sasl="off" sasl_password="" sasl_username="" tls_client_auth="0" tls="off" client_tls_cert="" client_tls_key="" brokers="172.17.0.11:29093" topic="my-topic" version="" --insecure
Successfully applied new settings.
Please restart your server 'mc admin service restart myminio'.
  1. Restart MinIO:
mc admin service restart myminio --insecure
  1. Enable Kafka bucket notification using MinIO client
mc mb myminio/images --insecure
mc event add  myminio/images arn:minio:sqs::1:kafka --suffix .jpg --insecure
mc event list myminio/images --insecure

You should see:

$ mc mb myminio/images --insecure
Bucket created successfully `myminio/images`.
$ mc event add  myminio/images arn:minio:sqs::1:kafka --suffix .jpg --insecure
Successfully added arn:minio:sqs::1:kafka
$ mc event list myminio/images --insecure
arn:minio:sqs::1:kafka   s3:ObjectCreated:*,s3:ObjectRemoved:*,s3:ObjectAccessed:*   Filter: suffix=".jpg"
  1. Upload/copy image to folder created
mc cp rose.jpg myminio/images --insecure
  1. As a result, the consumer from step 7 above will print the event:
{
  "EventName": "s3:ObjectCreated:Put",
  "Key": "images/rose.jpg",
  "Records": [
    {
      "eventVersion": "2.0",
      "eventSource": "minio:s3",
      "awsRegion": "",
      "eventTime": "2022-04-05T01:43:53.018Z",
      "eventName": "s3:ObjectCreated:Put",
      "userIdentity": {
        "principalId": "minio"
      },
      "requestParameters": {
        "principalId": "minio",
        "region": "",
        "sourceIPAddress": "127.0.0.1"
      },
      "responseElements": {
        "content-length": "0",
        "x-amz-request-id": "16E2DCA93D259AE4",
        "x-minio-deployment-id": "968b47d7-4857-4cf0-8906-58ce4716e1e6",
        "x-minio-origin-endpoint": "https://minio.tenant-lite.svc.cluster.local"
      },
      "s3": {
        "s3SchemaVersion": "1.0",
        "configurationId": "Config",
        "bucket": {
          "name": "images",
          "ownerIdentity": {
            "principalId": "minio"
          },
          "arn": "arn:aws:s3:::images"
        },
        "object": {
          "key": "rose.jpg",
          "size": 165352,
          "eTag": "c8e032a8b653aebb6b6141d50b5f3cd3",
          "contentType": "image/jpeg",
          "userMetadata": {
            "content-type": "image/jpeg"
          },
          "sequencer": "16E2DCA93DFB0DCC"
        }
      },
      "source": {
        "host": "127.0.0.1",
        "port": "",
        "userAgent": "MinIO (darwin; arm64) minio-go/v7.0.23 mc/RELEASE.2022-02-23T03-15-59Z"
      }
    }
  ]
}

Using OpenShift

Please make sure to use x86 architecture and follow steps from RedHat video below or similar:

https://www.youtube.com/watch?v=fYVioaEx2HY

Steps:

  1. Start cluster with CodeReady Containers:
crc start

You should see:

Started the OpenShift cluster.

The server is accessible via web console at:
  https://console-openshift-console.apps-crc.testing

Log in as administrator:
  Username: kubeadmin
  Password: 2yajb-6Kz5j-5ijRu-cSiKd

Log in as user:
  Username: developer
  Password: developer

Use the 'oc' command line interface:
  $ eval $(crc oc-env)
  $ oc login -u developer https://api.crc.testing:6443
  1. Install cert-manager:
kubectl create -f https://github.com/jetstack/cert-manager/releases/download/v1.6.2/cert-manager.yaml
Clone this wiki locally