Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] pod in CrashLoopBackOff status after creating kafka cluster with the latest 1.0 yaml #8348

Closed
tianyue86 opened this issue Oct 29, 2024 · 1 comment · Fixed by apecloud/kubeblocks-addons#1159
Assignees
Labels
kind/bug Something isn't working
Milestone

Comments

@tianyue86
Copy link

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

  1. Get the latest kafka cluster yaml
tianyue@192 kbcli % helm template kafkaclu02 kubeblocks-addons/kafka-cluster --version 1.0.0-alpha.0
---
# Source: kafka-cluster/templates/cluster.yaml
apiVersion: apps.kubeblocks.io/v1
kind: Cluster
metadata:
  name: kafkaclu02
  labels: 
    helm.sh/chart: kafka-cluster-1.0.0-alpha.0
    app.kubernetes.io/version: "3.3.2"
    app.kubernetes.io/instance: kafkaclu02
  annotations:
    "kubeblocks.io/extra-env": '{"KB_KAFKA_ENABLE_SASL":"false","KB_KAFKA_BROKER_HEAP":"-XshowSettings:vm -XX:MaxRAMPercentage=100 -Ddepth=64","KB_KAFKA_CONTROLLER_HEAP":"-XshowSettings:vm -XX:MaxRAMPercentage=100 -Ddepth=64","KB_KAFKA_PUBLIC_ACCESS":"false"}'

spec:
  clusterDef: kafka # ref clusterdefinition.name
  terminationPolicy: Delete
  topology:
        combined_monitor
  componentSpecs:
    - name: kafka-combine
      tls: false
      replicas: 1
      serviceAccountName: 
      services:
        - name: advertised-listener
          serviceType: ClusterIP
          podService: true      
      resources:
        limits:
          cpu: "0.5"
          memory: "0.5Gi"
        requests:
          cpu: "0.5"
          memory: "0.5Gi"
    - name: kafka-exporter
      replicas: 1
      
      resources:
        limits:
          cpu: "0.5"
          memory: "0.5Gi"
        requests:
          cpu: "0.5"
          memory: "0.5Gi"
  1. Apply yaml to create cluster
tianyue@192 kbcli % k apply -f kafkacn.yaml
cluster.apps.kubeblocks.io/kafkaclu02 created
  1. Check the cluster status: Abnormal
  2. Describe cluster
tianyue@192 kbcli % k describe cluster kafkaclu02
Name:         kafkaclu02
Namespace:    default
Labels:       app.kubernetes.io/instance=kafkaclu02
              app.kubernetes.io/version=3.3.2
              clusterdefinition.kubeblocks.io/name=kafka
              helm.sh/chart=kafka-cluster-1.0.0-alpha.0
Annotations:  kubeblocks.io/extra-env:
                {"KB_KAFKA_ENABLE_SASL":"false","KB_KAFKA_BROKER_HEAP":"-XshowSettings:vm -XX:MaxRAMPercentage=100 -Ddepth=64","KB_KAFKA_CONTROLLER_HEAP":...
API Version:  apps.kubeblocks.io/v1
Kind:         Cluster
Metadata:
  Creation Timestamp:  2024-10-29T07:52:35Z
  Finalizers:
    cluster.kubeblocks.io/finalizer
  Generation:        2
  Resource Version:  34527198
  UID:               dcb5b926-88e6-49f5-a404-827b31083bb5
Spec:
  Cluster Def:  kafka
  Component Specs:
    Component Def:  kafka-combine
    Name:           kafka-combine
    Replicas:       1
    Resources:
      Limits:
        Cpu:     500m
        Memory:  512Mi
      Requests:
        Cpu:          500m
        Memory:       512Mi
    Service Version:  3.3.2
    Services:
      Name:          advertised-listener
      Pod Service:   true
      Service Type:  ClusterIP
    Component Def:   kafka-exporter
    Name:            kafka-exporter
    Replicas:        1
    Resources:
      Limits:
        Cpu:     500m
        Memory:  512Mi
      Requests:
        Cpu:           500m
        Memory:        512Mi
    Service Version:   1.6.0
  Termination Policy:  Delete
  Topology:            combined_monitor
Status:
  Components:
    Kafka - Combine:
      Message:
        InstanceSet/kafkaclu02-kafka-combine:  ["kafkaclu02-kafka-combine-0"]
      Phase:                                   Failed
    Kafka - Exporter:
      Message:
        Reason:  the component to be created
  Conditions:
    Last Transition Time:  2024-10-29T07:52:35Z
    Message:               The operator has started the provisioning of Cluster: kafkaclu02
    Observed Generation:   2
    Reason:                PreCheckSucceed
    Status:                True
    Type:                  ProvisioningStarted
    Last Transition Time:  2024-10-29T07:52:35Z
    Message:               Successfully applied for resources
    Observed Generation:   2
    Reason:                ApplyResourcesSucceed
    Status:                True
    Type:                  ApplyResources
    Last Transition Time:  2024-10-29T07:52:59Z
    Message:               cluster kafkaclu02 is NOT ready, unavailable components: kafka-combine
    Reason:                ComponentsNotReady
    Status:                False
    Type:                  Ready
  Observed Generation:     2
  Phase:                   Abnormal
Events:
  Type     Reason                    Age                   From                Message
  ----     ------                    ----                  ----                -------
  Normal   PreCheckSucceed           36m (x2 over 36m)     cluster-controller  The operator has started the provisioning of Cluster: kafkaclu02
  Normal   ApplyResourcesSucceed     36m (x2 over 36m)     cluster-controller  Successfully applied for resources
  Normal   ComponentPhaseTransition  36m (x4 over 36m)     cluster-controller  cluster component kafka-combine is Creating
  Warning  Abnormal                  36m                   cluster-controller  Cluster: kafkaclu02 is Abnormal, check according to the components message
  Normal   ComponentPhaseTransition  15m (x12 over 36m)    cluster-controller  cluster component kafka-combine is Failed
  Normal   ComponentPhaseTransition  4m59s (x10 over 36m)  cluster-controller  cluster component kafka-combine is Updating
  1. Describe pod - see error:
kafkaclu02-kafka-combine-0                                        1/2     CrashLoopBackOff   8 (3m15s ago)   20m
tianyue@192 kbcli % k describe pod kafkaclu02-kafka-combine-0
Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Scheduled  20m                  default-scheduler  Successfully assigned default/kafkaclu02-kafka-combine-0 to ip-172-31-7-55.ap-northeast-1.compute.internal
  Normal   Pulling    20m                  kubelet            Pulling image "docker.io/apecloud/kafka:3.3.2-debian-11-r54"
  Normal   Pulled     20m                  kubelet            Successfully pulled image "docker.io/apecloud/kafka:3.3.2-debian-11-r54" in 11.629s (11.629s including waiting). Image size: 351351482 bytes.
  Normal   Pulling    20m                  kubelet            Pulling image "docker.io/apecloud/jmx-exporter:0.18.0-debian-11-r20"
  Normal   Pulled     20m                  kubelet            Successfully pulled image "docker.io/apecloud/jmx-exporter:0.18.0-debian-11-r20" in 9.643s (9.643s including waiting). Image size: 241736233 bytes.
  Normal   Created    20m                  kubelet            Created container jmx-exporter
  Normal   Started    20m                  kubelet            Started container jmx-exporter
  Normal   Created    19m (x4 over 20m)    kubelet            Created container kafka
  Normal   Started    19m (x4 over 20m)    kubelet            Started container kafka
  Normal   Pulled     19m (x3 over 20m)    kubelet            Container image "docker.io/apecloud/kafka:3.3.2-debian-11-r54" already present on machine
  Warning  BackOff    22s (x102 over 19m)  kubelet            Back-off restarting failed container kafka in pod kafkaclu02-kafka-combine-0_default(82edadba-7e14-4d45-84c4-fa9fd0701a95)

Other check:
addon:
kafka 1.0.0-alpha.0 community Enabled true

tianyue@192 kbcli % k get cmpd
NAME SERVICE SERVICE-VERSION STATUS AGE
apecloud-mysql mysql 8.0.30 Available 24h
etcd etcd v3.5.15 Available 24h
kafka-broker kafka 3.3.2 Available 24h
kafka-combine kafka 3.3.2 Available 24h
kafka-controller kafka-controller 3.3.2 Available 24h
kafka-exporter kafka-exporter 1.6.0 Available 24h

@tianyue86 tianyue86 added the kind/bug Something isn't working label Oct 29, 2024
@humingmingst
Copy link
Contributor

I’d like to give this a try.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants