Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] pulsar cluster hscale broker pod crash, upgrade KB from 0.7.5 to 0.8.3 #7048

Open
JashBook opened this issue Apr 15, 2024 · 1 comment
Open

[BUG] pulsar cluster hscale broker pod crash, upgrade KB from 0.7.5 to 0.8.3 #7048

JashBook opened this issue Apr 15, 2024 · 1 comment
Assignees
@caiq1nyu
Labels
area/upgrade kind/bug Something isn't working
Milestone
Release 0.8.5

Comments

@JashBook
Copy link
Collaborator

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

  1. instal KB 0.7.5
  2. create pulsar cluster
apiVersion: apps.kubeblocks.io/v1alpha1
kind: Cluster
metadata:
  name: pulsar-upkb781
  namespace: default
spec:
  affinity:
    podAntiAffinity: Preferred
    tenancy: SharedNode
    topologyKeys:
    - kubernetes.io/hostname
  clusterDefinitionRef: pulsar
  clusterVersionRef: pulsar-2.11.2
  componentSpecs:
  - classDefRef:
      class: ""
    componentDefRef: pulsar-broker
    monitor: true
    name: pulsar-broker
    noCreatePDB: false
    replicas: 1
    resources:
      limits:
        cpu: 600m
        memory: 644245094400m
      requests:
        cpu: 600m
        memory: 644245094400m
    rsmTransformPolicy: ToSts
  - componentDefRef: bookies
    monitor: true
    name: bookies
    noCreatePDB: false
    replicas: 3
    resources:
      limits:
        cpu: 500m
        memory: 512Mi
      requests:
        cpu: 500m
        memory: 512Mi
    rsmTransformPolicy: ToSts
    volumeClaimTemplates:
    - name: ledgers
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
    - name: journal
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
  - componentDefRef: zookeeper
    monitor: true
    name: zookeeper
    noCreatePDB: false
    replicas: 3
    resources:
      limits:
        cpu: 100m
        memory: 512Mi
      requests:
        cpu: 100m
        memory: 512Mi
    rsmTransformPolicy: ToSts
    volumeClaimTemplates:
    - name: data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
  - componentDefRef: pulsar-proxy
    monitor: true
    name: pulsar-proxy
    noCreatePDB: false
    replicas: 1
    resources:
      limits:
        cpu: 100m
        memory: 512Mi
      requests:
        cpu: 100m
        memory: 512Mi
    rsmTransformPolicy: ToSts
  monitor: {}
  resources:
    cpu: "0"
    memory: "0"
  storage:
    size: "0"
  terminationPolicy: WipeOut
  1. hscale out broker 1 --> 2 success
kbcli_0.7.5 cluster hscale pulsar-upkb781 --components pulsar-broker --replicas 2 --auto-approve
  1. upgrade KB from 0.7.5 to 0.8.3
  2. hscale out broker 1 --> 2 failure
kbcli cluster hscale pulsar-upkb781 --components pulsar-broker --replicas 3 --auto-approve
  1. see error
➜  ~ kubectl get cluster 
NAME             CLUSTER-DEFINITION   VERSION         TERMINATION-POLICY   STATUS     AGE
pulsar-upkb781   pulsar               pulsar-2.11.2   WipeOut              Updating   74m
➜  ~ 
➜  ~ kubectl get pod 
NAME                                            READY   STATUS             RESTARTS        AGE
kb-addon-csi-hostpath-driver-0                  8/8     Running            0               82m
kb-addon-kubebench-bb5979455-64r25              1/1     Running            0               43m
kb-addon-minio-79f868bbf9-zzjbr                 1/1     Running            0               82m
kb-addon-snapshot-controller-8484bbd44c-k7jvx   1/1     Running            0               83m
kubeblocks-6d4987bff6-4nl8s                     1/1     Running            0               50m
kubeblocks-dataprotection-fb6dc4894-bsx85       1/1     Running            0               50m
pulsar-upkb781-bookies-0                        2/2     Running            0               9m36s
pulsar-upkb781-bookies-1                        2/2     Running            0               10m
pulsar-upkb781-bookies-2                        2/2     Running            0               10m
pulsar-upkb781-pulsar-broker-0                  3/3     Running            0               58m
pulsar-upkb781-pulsar-broker-1                  3/3     Running            0               60m
pulsar-upkb781-pulsar-broker-2                  2/3     CrashLoopBackOff   6 (4m33s ago)   10m
pulsar-upkb781-pulsar-proxy-0                   2/2     Running            0               60m
pulsar-upkb781-zookeeper-0                      2/2     Running            0               60m
pulsar-upkb781-zookeeper-1                      2/2     Running            0               60m
pulsar-upkb781-zookeeper-2                      2/2     Running            0               60m
➜  ~ 
➜  ~ kubectl get ops
NAME                                     TYPE                CLUSTER          STATUS    PROGRESS   AGE
pulsar-upkb781-horizontalscaling-tncc8   HorizontalScaling   pulsar-upkb781   Running   0/1        11m

describe pod

kubectl describe pod pulsar-upkb781-pulsar-broker-2 
Name:         pulsar-upkb781-pulsar-broker-2
Namespace:    default
Priority:     0
Node:         minikube/192.168.49.2
Start Time:   Mon, 15 Apr 2024 11:23:08 +0800
Labels:       app.kubernetes.io/component=pulsar-broker
              app.kubernetes.io/instance=pulsar-upkb781
              app.kubernetes.io/managed-by=kubeblocks
              app.kubernetes.io/name=pulsar
              app.kubernetes.io/version=pulsar-2.11.2
              apps.kubeblocks.io/cluster-uid=398b3dda-5d92-4970-8a8f-ac89ca1da2ab
              apps.kubeblocks.io/component-name=pulsar-broker
              apps.kubeblocks.io/workload-type=Stateful
              clusterdefinition.kubeblocks.io/name=pulsar
              clusterversion.kubeblocks.io/name=pulsar-2.11.2
              controller-revision-hash=pulsar-upkb781-pulsar-broker-f4689f9dc
              statefulset.kubernetes.io/pod-name=pulsar-upkb781-pulsar-broker-2
Annotations:  apps.kubeblocks.io/component-replicas: 3
              config.kubeblocks.io/restart-broker-config: fdd4688
              kubeblocks.io/restart: 2024-04-15T02:33:18Z
Status:       Running
IP:           10.244.18.26
IPs:
  IP:           10.244.18.26
Controlled By:  StatefulSet/pulsar-upkb781-pulsar-broker
Init Containers:
  init-broker-cluster:
    Container ID:  docker://4343a0249b1cb2948ab9c3607e2fc176568835a0ca858da362b6032cf9e589da
    Image:         infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/pulsar:2.11.2
    Image ID:      docker-pullable://infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/pulsar@sha256:d394d32706c927c54b28a2e0f051ac7ee8ac25bdb69a3fb23d7de6510e4daa1a
    Port:          <none>
    Host Port:     <none>
    Command:
      /kb-scripts/init-broker.sh
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Mon, 15 Apr 2024 11:23:09 +0800
      Finished:     Mon, 15 Apr 2024 11:23:09 +0800
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     0
      memory:  0
    Requests:
      cpu:     0
      memory:  0
    Environment Variables from:
      pulsar-upkb781-pulsar-broker-env                 ConfigMap  Optional: false
      pulsar-upkb781-pulsar-broker-broker-env-envfrom  ConfigMap  Optional: false
    Environment:
      KB_POD_NAME:               pulsar-upkb781-pulsar-broker-2 (v1:metadata.name)
      KB_POD_UID:                 (v1:metadata.uid)
      KB_NAMESPACE:              default (v1:metadata.namespace)
      KB_SA_NAME:                 (v1:spec.serviceAccountName)
      KB_NODENAME:                (v1:spec.nodeName)
      KB_HOST_IP:                 (v1:status.hostIP)
      KB_POD_IP:                  (v1:status.podIP)
      KB_POD_IPS:                 (v1:status.podIPs)
      KB_HOSTIP:                  (v1:status.hostIP)
      KB_PODIP:                   (v1:status.podIP)
      KB_PODIPS:                  (v1:status.podIPs)
      KB_CLUSTER_NAME:           pulsar-upkb781
      KB_COMP_NAME:              pulsar-broker
      KB_CLUSTER_COMP_NAME:      pulsar-upkb781-pulsar-broker
      KB_CLUSTER_UID_POSTFIX_8:  ca1da2ab
      KB_POD_FQDN:               $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
      brokerServiceUrl:          pulsar://$(KB_CLUSTER_COMP_NAME).$(KB_NAMESPACE).svc.cluster.local:6650
      clusterName:               $(KB_NAMESPACE)-$(KB_CLUSTER_COMP_NAME)
      webServiceUrl:             http://$(KB_CLUSTER_COMP_NAME).$(KB_NAMESPACE).svc.cluster.local:80
    Mounts:
      /kb-scripts from scripts (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-kc8n9 (ro)
  init-sysctl:
    Container ID:  docker://921f20da52120a5d491f7947d077169883475db3684cd526471767dcdd2ffe20
    Image:         infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/pulsar:2.11.2
    Image ID:      docker-pullable://infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/pulsar@sha256:d394d32706c927c54b28a2e0f051ac7ee8ac25bdb69a3fb23d7de6510e4daa1a
    Port:          <none>
    Host Port:     <none>
    Command:
      /kb-scripts/init-broker-sysctl.sh
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Mon, 15 Apr 2024 11:23:10 +0800
      Finished:     Mon, 15 Apr 2024 11:23:10 +0800
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     0
      memory:  0
    Requests:
      cpu:     0
      memory:  0
    Environment Variables from:
      pulsar-upkb781-pulsar-broker-env  ConfigMap  Optional: false
    Environment:
      KB_POD_NAME:               pulsar-upkb781-pulsar-broker-2 (v1:metadata.name)
      KB_POD_UID:                 (v1:metadata.uid)
      KB_NAMESPACE:              default (v1:metadata.namespace)
      KB_SA_NAME:                 (v1:spec.serviceAccountName)
      KB_NODENAME:                (v1:spec.nodeName)
      KB_HOST_IP:                 (v1:status.hostIP)
      KB_POD_IP:                  (v1:status.podIP)
      KB_POD_IPS:                 (v1:status.podIPs)
      KB_HOSTIP:                  (v1:status.hostIP)
      KB_PODIP:                   (v1:status.podIP)
      KB_PODIPS:                  (v1:status.podIPs)
      KB_CLUSTER_NAME:           pulsar-upkb781
      KB_COMP_NAME:              pulsar-broker
      KB_CLUSTER_COMP_NAME:      pulsar-upkb781-pulsar-broker
      KB_CLUSTER_UID_POSTFIX_8:  ca1da2ab
      KB_POD_FQDN:               $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
    Mounts:
      /kb-scripts from scripts (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-kc8n9 (ro)
  init-pulsar-tools:
    Container ID:  docker://d8fb905a422daf699cdd16783dd736bf7ab02423de9b7a35ff067c355799ceb3
    Image:         infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/pulsartools:2.11
    Image ID:      docker-pullable://infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/pulsartools@sha256:ca94a8331febfe3a51b79ffe3c366a554e708c9a607fb9f6c0539cc28914a49a
    Port:          <none>
    Host Port:     <none>
    Command:
      cp
      /bin/pulsar-admin
      /kb/tools/pulsar-admin
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Mon, 15 Apr 2024 11:23:11 +0800
      Finished:     Mon, 15 Apr 2024 11:23:11 +0800
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     0
      memory:  0
    Requests:
      cpu:     0
      memory:  0
    Environment Variables from:
      pulsar-upkb781-pulsar-broker-env  ConfigMap  Optional: false
    Environment:
      KB_POD_NAME:               pulsar-upkb781-pulsar-broker-2 (v1:metadata.name)
      KB_POD_UID:                 (v1:metadata.uid)
      KB_NAMESPACE:              default (v1:metadata.namespace)
      KB_SA_NAME:                 (v1:spec.serviceAccountName)
      KB_NODENAME:                (v1:spec.nodeName)
      KB_HOST_IP:                 (v1:status.hostIP)
      KB_POD_IP:                  (v1:status.podIP)
      KB_POD_IPS:                 (v1:status.podIPs)
      KB_HOSTIP:                  (v1:status.hostIP)
      KB_PODIP:                   (v1:status.podIP)
      KB_PODIPS:                  (v1:status.podIPs)
      KB_CLUSTER_NAME:           pulsar-upkb781
      KB_COMP_NAME:              pulsar-broker
      KB_CLUSTER_COMP_NAME:      pulsar-upkb781-pulsar-broker
      KB_CLUSTER_UID_POSTFIX_8:  ca1da2ab
      KB_POD_FQDN:               $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
      TOOLS_SCRIPTS_PATH:        /opt/kb-tools/reload/broker-config
    Mounts:
      /kb/tools from kb-tools (rw)
      /opt/config-manager from config-manager-config (rw)
      /opt/kb-tools/reload/broker-config from cm-script-broker-config (rw)
      /opt/pulsar/conf from pulsar-config (rw)
      /opt/pulsar/none from broker-env (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-kc8n9 (ro)
  init-pulsar-client-config:
    Container ID:  docker://bcdc71eb4df5beab34d2dbccc3007c2e038f6205e1fc383ca661f901a28d6281
    Image:         infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/pulsar:2.11.2
    Image ID:      docker-pullable://infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/pulsar@sha256:d394d32706c927c54b28a2e0f051ac7ee8ac25bdb69a3fb23d7de6510e4daa1a
    Port:          <none>
    Host Port:     <none>
    Command:
      $(TOOLS_SCRIPTS_PATH)/install-pulsar-client-config.sh
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Mon, 15 Apr 2024 11:23:12 +0800
      Finished:     Mon, 15 Apr 2024 11:23:12 +0800
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     0
      memory:  0
    Requests:
      cpu:     0
      memory:  0
    Environment Variables from:
      pulsar-upkb781-pulsar-broker-env                 ConfigMap  Optional: false
      pulsar-upkb781-pulsar-broker-broker-env-envfrom  ConfigMap  Optional: false
    Environment:
      KB_POD_NAME:               pulsar-upkb781-pulsar-broker-2 (v1:metadata.name)
      KB_POD_UID:                 (v1:metadata.uid)
      KB_NAMESPACE:              default (v1:metadata.namespace)
      KB_SA_NAME:                 (v1:spec.serviceAccountName)
      KB_NODENAME:                (v1:spec.nodeName)
      KB_HOST_IP:                 (v1:status.hostIP)
      KB_POD_IP:                  (v1:status.podIP)
      KB_POD_IPS:                 (v1:status.podIPs)
      KB_HOSTIP:                  (v1:status.hostIP)
      KB_PODIP:                   (v1:status.podIP)
      KB_PODIPS:                  (v1:status.podIPs)
      KB_CLUSTER_NAME:           pulsar-upkb781
      KB_COMP_NAME:              pulsar-broker
      KB_CLUSTER_COMP_NAME:      pulsar-upkb781-pulsar-broker
      KB_CLUSTER_UID_POSTFIX_8:  ca1da2ab
      KB_POD_FQDN:               $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
      TOOLS_SCRIPTS_PATH:        /opt/kb-tools/reload/broker-config
    Mounts:
      /kb/tools from kb-tools (rw)
      /opt/config-manager from config-manager-config (rw)
      /opt/kb-tools/reload/broker-config from cm-script-broker-config (rw)
      /opt/pulsar/conf from pulsar-config (rw)
      /opt/pulsar/none from broker-env (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-kc8n9 (ro)
Containers:
  broker:
    Container ID:  docker://ce24b1766308865fd8a4f301caead39d809676dea1d037929e03e48bf5cc92c5
    Image:         infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/pulsar:2.11.2
    Image ID:      docker-pullable://infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/pulsar@sha256:d394d32706c927c54b28a2e0f051ac7ee8ac25bdb69a3fb23d7de6510e4daa1a
    Ports:         6650/TCP, 8080/TCP, 9092/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP
    Command:
      sh
      -c
    Args:
      set -x
      /kb-scripts/merge_pulsar_config.py conf/client.conf /opt/pulsar/conf/client.conf && \
      /kb-scripts/merge_pulsar_config.py conf/broker.conf /opt/pulsar/conf/broker.conf && \
      bin/apply-config-from-env.py conf/broker.conf && \
      bin/apply-config-from-env.py conf/client.conf && \
      echo 'OK' > status;exec bin/pulsar broker
      
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Mon, 15 Apr 2024 11:29:30 +0800
      Finished:     Mon, 15 Apr 2024 11:29:34 +0800
    Ready:          False
    Restart Count:  6
    Limits:
      cpu:     600m
      memory:  644245094400m
    Requests:
      cpu:      600m
      memory:   644245094400m
    Liveness:   http-get http://:http/status.html delay=5s timeout=100s period=10s #success=1 #failure=30
    Readiness:  http-get http://:http/status.html delay=30s timeout=100s period=10s #success=1 #failure=3
    Startup:    http-get http://:http/status.html delay=5s timeout=100s period=10s #success=1 #failure=30
    Environment Variables from:
      pulsar-upkb781-pulsar-broker-env                 ConfigMap  Optional: false
      pulsar-upkb781-pulsar-broker-broker-env-envfrom  ConfigMap  Optional: false
      pulsar-upkb781-pulsar-broker-rsm-env             ConfigMap  Optional: false
    Environment:
      KB_POD_NAME:                               pulsar-upkb781-pulsar-broker-2 (v1:metadata.name)
      KB_POD_UID:                                 (v1:metadata.uid)
      KB_NAMESPACE:                              default (v1:metadata.namespace)
      KB_SA_NAME:                                 (v1:spec.serviceAccountName)
      KB_NODENAME:                                (v1:spec.nodeName)
      KB_HOST_IP:                                 (v1:status.hostIP)
      KB_POD_IP:                                  (v1:status.podIP)
      KB_POD_IPS:                                 (v1:status.podIPs)
      KB_HOSTIP:                                  (v1:status.hostIP)
      KB_PODIP:                                   (v1:status.podIP)
      KB_PODIPS:                                  (v1:status.podIPs)
      KB_CLUSTER_NAME:                           pulsar-upkb781
      KB_COMP_NAME:                              pulsar-broker
      KB_CLUSTER_COMP_NAME:                      pulsar-upkb781-pulsar-broker
      KB_CLUSTER_UID_POSTFIX_8:                  ca1da2ab
      KB_POD_FQDN:                               $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
      SERVICE_PORT:                              8080
      POD_NAME:                                  $(KB_POD_NAME)
      PULSAR_PREFIX_brokerServicePort:           6650
      PULSAR_PREFIX_internalListenerName:        cluster
      PULSAR_PREFIX_advertisedListeners:         cluster:pulsar://$(POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc.cluster.local:6650
      PULSAR_PREFIX_allowAutoTopicCreationType:  partitioned
      PULSAR_PREFIX_kafkaListeners:              PLAINTEXT://0.0.0.0:9092
      PULSAR_PREFIX_kafkaAdvertisedListeners:    PLAINTEXT://$(POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc.cluster.local:9092
      brokerServiceUrl:                          pulsar://$(KB_CLUSTER_COMP_NAME).$(KB_NAMESPACE).svc.cluster.local:6650
      clusterName:                               $(KB_NAMESPACE)-$(KB_CLUSTER_COMP_NAME)
      webServiceUrl:                             http://$(KB_CLUSTER_COMP_NAME).$(KB_NAMESPACE).svc.cluster.local:80
    Mounts:
      /kb-scripts from scripts (rw)
      /kb/tools from kb-tools (rw)
      /opt/pulsar/conf from pulsar-config (rw)
      /opt/pulsar/none from broker-env (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-kc8n9 (ro)
  metrics:
    Container ID:  docker://4394fadfcfed2759cb30e35bc0ec8762011db5e51ea12b59476d8e03985173f4
    Image:         infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/agamotto:0.1.2-beta.2
    Image ID:      docker-pullable://infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/agamotto@sha256:a85d7d80d33de188ab39880d8d6013b663ebade9186371ce09a06c29dfa443ac
    Port:          1234/TCP
    Host Port:     0/TCP
    Command:
      /bin/agamotto
      --config=/opt/agamotto/agamotto-config.yaml
      --feature-gates=-pkg.translator.prometheus.NormalizeName
    State:          Running
      Started:      Mon, 15 Apr 2024 11:23:14 +0800
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     0
      memory:  0
    Requests:
      cpu:     0
      memory:  0
    Environment Variables from:
      pulsar-upkb781-pulsar-broker-env      ConfigMap  Optional: false
      pulsar-upkb781-pulsar-broker-rsm-env  ConfigMap  Optional: false
    Environment:
      KB_POD_NAME:               pulsar-upkb781-pulsar-broker-2 (v1:metadata.name)
      KB_POD_UID:                 (v1:metadata.uid)
      KB_NAMESPACE:              default (v1:metadata.namespace)
      KB_SA_NAME:                 (v1:spec.serviceAccountName)
      KB_NODENAME:                (v1:spec.nodeName)
      KB_HOST_IP:                 (v1:status.hostIP)
      KB_POD_IP:                  (v1:status.podIP)
      KB_POD_IPS:                 (v1:status.podIPs)
      KB_HOSTIP:                  (v1:status.hostIP)
      KB_PODIP:                   (v1:status.podIP)
      KB_PODIPS:                  (v1:status.podIPs)
      KB_CLUSTER_NAME:           pulsar-upkb781
      KB_COMP_NAME:              pulsar-broker
      KB_CLUSTER_COMP_NAME:      pulsar-upkb781-pulsar-broker
      KB_CLUSTER_UID_POSTFIX_8:  ca1da2ab
      KB_POD_FQDN:               $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
      JOB_NAME:                  broker
      SERVICE_PORT:              8080
      POD_NAME:                  $(KB_POD_NAME)
      POD_NAMESPACE:             $(KB_NAMESPACE)
      LOG_LEVEL:                 info
    Mounts:
      /opt/agamotto from agamotto-configuration (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-kc8n9 (ro)
  config-manager:
    Container ID:  docker://7196ad355b7140b0b90d33661799896b90490e733969cc60c09fa8bbad4258ce
    Image:         infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/kubeblocks-tools:0.8.3-beta.7
    Image ID:      docker-pullable://infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/kubeblocks-tools@sha256:d5c966a4f96c5e60f7530fb08c55a40d30e501ca05e41e48574be3e4ab7ec220
    Port:          <none>
    Host Port:     <none>
    Command:
      env
    Args:
      PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:$(TOOLS_PATH)
      /bin/reloader
      --log-level
      info
      --operator-update-enable
      --tcp
      9901
      --config
      /opt/config-manager/config-manager.yaml
    State:          Running
      Started:      Mon, 15 Apr 2024 11:23:14 +0800
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     0
      memory:  0
    Requests:
      cpu:     0
      memory:  0
    Environment Variables from:
      pulsar-upkb781-pulsar-broker-env      ConfigMap  Optional: false
      pulsar-upkb781-pulsar-broker-rsm-env  ConfigMap  Optional: false
    Environment:
      KB_POD_NAME:               pulsar-upkb781-pulsar-broker-2 (v1:metadata.name)
      KB_POD_UID:                 (v1:metadata.uid)
      KB_NAMESPACE:              default (v1:metadata.namespace)
      KB_SA_NAME:                 (v1:spec.serviceAccountName)
      KB_NODENAME:                (v1:spec.nodeName)
      KB_HOST_IP:                 (v1:status.hostIP)
      KB_POD_IP:                  (v1:status.podIP)
      KB_POD_IPS:                 (v1:status.podIPs)
      KB_HOSTIP:                  (v1:status.hostIP)
      KB_PODIP:                   (v1:status.podIP)
      KB_PODIPS:                  (v1:status.podIPs)
      KB_CLUSTER_NAME:           pulsar-upkb781
      KB_COMP_NAME:              pulsar-broker
      KB_CLUSTER_COMP_NAME:      pulsar-upkb781-pulsar-broker
      KB_CLUSTER_UID_POSTFIX_8:  ca1da2ab
      KB_POD_FQDN:               $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
      CONFIG_MANAGER_POD_IP:      (v1:status.podIP)
      DB_TYPE:                   pulsar-broker
      TOOLS_PATH:                /opt/kb-tools/reload/broker-config:/opt/config-manager:/kb/tools
    Mounts:
      /kb/tools from kb-tools (rw)
      /opt/config-manager from config-manager-config (rw)
      /opt/kb-tools/reload/broker-config from cm-script-broker-config (rw)
      /opt/pulsar/conf from pulsar-config (rw)
      /opt/pulsar/none from broker-env (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-kc8n9 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  agamotto-configuration:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      pulsar-upkb781-pulsar-broker-agamotto-configuration
    Optional:  false
  pulsar-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      pulsar-upkb781-pulsar-broker-broker-config
    Optional:  false
  broker-env:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      pulsar-upkb781-pulsar-broker-broker-env
    Optional:  false
  scripts:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      pulsar-upkb781-pulsar-broker-pulsar-scripts
    Optional:  false
  cm-script-broker-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      sidecar-pulsar-tools-script-pulsar-upkb781
    Optional:  false
  config-manager-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      sidecar-pulsar-upkb781-pulsar-broker-config-manager-config
    Optional:  false
  kb-tools:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  kube-api-access-kc8n9:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 kb-data=true:NoSchedule
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                 From               Message
  ----     ------     ----                ----               -------
  Normal   Scheduled  11m                 default-scheduler  Successfully assigned default/pulsar-upkb781-pulsar-broker-2 to minikube
  Normal   Pulled     11m                 kubelet            Container image "infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/pulsar:2.11.2" already present on machine
  Normal   Created    11m                 kubelet            Created container init-broker-cluster
  Normal   Started    11m                 kubelet            Started container init-broker-cluster
  Normal   Pulled     11m                 kubelet            Container image "infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/pulsar:2.11.2" already present on machine
  Normal   Created    11m                 kubelet            Created container init-sysctl
  Normal   Started    11m                 kubelet            Started container init-sysctl
  Normal   Pulled     11m                 kubelet            Container image "infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/pulsartools:2.11" already present on machine
  Normal   Created    11m                 kubelet            Created container init-pulsar-tools
  Normal   Started    11m                 kubelet            Started container init-pulsar-tools
  Normal   Pulled     11m                 kubelet            Container image "infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/pulsar:2.11.2" already present on machine
  Normal   Created    11m                 kubelet            Created container init-pulsar-client-config
  Normal   Started    11m                 kubelet            Started container init-pulsar-client-config
  Normal   Pulled     11m                 kubelet            Container image "infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/agamotto:0.1.2-beta.2" already present on machine
  Normal   Created    11m                 kubelet            Created container metrics
  Normal   Pulled     11m                 kubelet            Container image "infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/kubeblocks-tools:0.8.3-beta.7" already present on machine
  Normal   Created    11m                 kubelet            Created container config-manager
  Normal   Started    11m                 kubelet            Started container config-manager
  Normal   Started    11m                 kubelet            Started container metrics
  Normal   Pulled     11m (x2 over 11m)   kubelet            Container image "infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/pulsar:2.11.2" already present on machine
  Normal   Created    11m (x2 over 11m)   kubelet            Created container broker
  Normal   Started    11m (x2 over 11m)   kubelet            Started container broker
  Warning  Unhealthy  11m                 kubelet            Startup probe failed: Get "http://10.244.18.26:8080/status.html": dial tcp 10.244.18.26:8080: connect: connection refused
  Warning  BackOff    71s (x56 over 11m)  kubelet            Back-off restarting failed container broker in pod pulsar-upkb781-pulsar-broker-2_default(9357606a-6f6b-4d01-b39b-b0bf48d7403b)

logs pod

➜  ~ kubectl logs pulsar-upkb781-pulsar-broker-2  broker
+ /kb-scripts/merge_pulsar_config.py conf/client.conf /opt/pulsar/conf/client.conf
Traceback (most recent call last):
  File "/kb-scripts/merge_pulsar_config.py", line 59, in <module>
    merge_config_files(dst_files, src_files)
  File "/kb-scripts/merge_pulsar_config.py", line 45, in merge_config_files
    lines, keys = load_config_file(src_file)
  File "/kb-scripts/merge_pulsar_config.py", line 11, in load_config_file
    for line in open(file):
FileNotFoundError: [Errno 2] No such file or directory: '/opt/pulsar/conf/client.conf'
+ exec bin/pulsar broker
VM settings:
    Max. Heap Size (Estimated): 186.00M
    Using VM: OpenJDK 64-Bit Server VM

2024-04-15T03:34:50,040Z [jdk.internal.loader.ClassLoaders$AppClassLoader@5bc2b487] error Uncaught exception in thread main: Required clusterName is null
java.lang.IllegalArgumentException: Required clusterName is null
	at org.apache.pulsar.common.configuration.PulsarConfigurationLoader.isComplete(PulsarConfigurationLoader.java:153)
	at org.apache.pulsar.PulsarBrokerStarter.loadConfig(PulsarBrokerStarter.java:72)
	at org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.<init>(PulsarBrokerStarter.java:169)
	at org.apache.pulsar.PulsarBrokerStarter.main(PulsarBrokerStarter.java:331)

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

  • OS: [e.g. iOS]
  • Browser [e.g. chrome, safari]
  • Version [e.g. 22]

Additional context
Add any other context about the problem here.
@JashBook JashBook added the kind/bug Something isn't working label Apr 15, 2024
@JashBook JashBook added this to the Release 0.8.3 milestone Apr 15, 2024
@ahjing99 ahjing99 added the severity/major Great chance user will encounter the same problem label Apr 15, 2024
@JashBook JashBook removed the severity/major Great chance user will encounter the same problem label Apr 15, 2024
@github-actions GitHub Actions
Copy link

This issue has been marked as stale because it has been open for 30 days with no activity

@github-actions github-actions bot added the Stale label May 20, 2024
@ahjing99 ahjing99 modified the milestones: Release 0.8.3, Release 0.8.4 May 24, 2024
@github-actions github-actions bot removed the Stale label May 27, 2024
@ahjing99 ahjing99 modified the milestones: Release 0.8.4, Release 0.8.5 Aug 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@caiq1nyu caiq1nyu

Labels
area/upgrade kind/bug Something isn't working
Projects
None yet
Milestone
Release 0.8.5
Development

No branches or pull requests
3 participants
@caiq1nyu @ahjing99 @JashBook