Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] kafka cluster execute ops Reconfiguring lead to kubeblocks CrashLoopBackOff #5612

Closed
linghan-hub opened this issue Oct 25, 2023 · 1 comment · Fixed by #5570
Closed

[BUG] kafka cluster execute ops Reconfiguring lead to kubeblocks CrashLoopBackOff #5612

linghan-hub opened this issue Oct 25, 2023 · 1 comment · Fixed by #5570
Assignees
@sophon-zt
Labels
bug kind/bug Something isn't working severity/major Great chance user will encounter the same problem
Milestone
Release 0.7.0

Comments

@linghan-hub
Copy link
Collaborator

linghan-hub commented Oct 25, 2023 •
edited
Loading 

kbcli version
Kubernetes: v1.25.14-eks-f8587cb
KubeBlocks: 0.7.0-beta.10
kbcli: 0.7.0-beta.10
  1. create kafka cluster
---
# Source: kafka-cluster/templates/rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kb-kafka-cluster
  namespace: default
  labels:
    helm.sh/chart: kafka-cluster-0.8.0-alpha.0
    app.kubernetes.io/version: "3.3.2"
    app.kubernetes.io/instance: kafka-cluster
---
# Source: kafka-cluster/templates/rbac.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kb-kafka-cluster
  labels:
    helm.sh/chart: kafka-cluster-0.8.0-alpha.0
    app.kubernetes.io/version: "3.3.2"
    app.kubernetes.io/instance: kafka-cluster
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kubeblocks-volume-protection-pod-role
subjects:
  - kind: ServiceAccount
    name: kb-kafka-cluster
    namespace: default
---
# Source: kafka-cluster/templates/rbac.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: kb-kafka-cluster
  labels:
    helm.sh/chart: kafka-cluster-0.8.0-alpha.0
    app.kubernetes.io/version: "3.3.2"
    app.kubernetes.io/instance: kafka-cluster
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kubeblocks-cluster-pod-role
subjects:
  - kind: ServiceAccount
    name: kb-kafka-cluster
    namespace: default
---
# Source: kafka-cluster/templates/cluster.yaml
apiVersion: apps.kubeblocks.io/v1alpha1
kind: Cluster
metadata:
  name: kafka-cluster
  labels: 
    helm.sh/chart: kafka-cluster-0.8.0-alpha.0
    app.kubernetes.io/version: "3.3.2"
    app.kubernetes.io/instance: kafka-cluster
  annotations:
    "kubeblocks.io/extra-env": '{"KB_KAFKA_ENABLE_SASL":"false","KB_KAFKA_BROKER_HEAP":"-XshowSettings:vm -XX:MaxRAMPercentage=100 -Ddepth=64","KB_KAFKA_CONTROLLER_HEAP":"-XshowSettings:vm -XX:MaxRAMPercentage=100 -Ddepth=64","KB_KAFKA_PUBLIC_ACCESS":"false"}'
spec:
  clusterDefinitionRef: kafka # ref clusterdefinition.name
  clusterVersionRef: kafka-3.3.2
  terminationPolicy: Delete  
  affinity:
    podAntiAffinity: Preferred
    topologyKeys:
      - kubernetes.io/hostname
    tenancy: SharedNode
  componentSpecs:
    - name: broker
      componentDefRef: kafka-server
      tls: false
      replicas: 1
      monitor: false
      serviceAccountName: kb-kafka-cluster      
      resources:
        limits:
          cpu: "0.5"
          memory: "0.5Gi"
        requests:
          cpu: "0.5"
          memory: "0.5Gi"      
      services:
  1. create ops
apiVersion: apps.kubeblocks.io/v1alpha1
kind: OpsRequest
metadata:
  generateName: kafka-cluster-reconfiguring-
spec:
  clusterRef: kafka-cluster
  reconfigure:
    componentName: broker
    configurations:
      - keys:
          - key: kafka-configuration-tpl
            parameters:
              - key: MaxRAMPercentage
                value: "200"
              - key: MaxGCPauseMillis
                value: "30"
        name: kafka-consensusset-config
  type: Reconfiguring
  1. see resource
k get cluster
NAME            CLUSTER-DEFINITION   VERSION       TERMINATION-POLICY   STATUS    AGE
kafka-cluster   kafka                kafka-3.3.2   Delete               Running   3m23s
k get pod
NAME                     READY   STATUS    RESTARTS   AGE
csi-attacher-s3-0        1/1     Running   0          46h
csi-provisioner-s3-0     2/2     Running   0          46h
csi-s3-2mmtt             2/2     Running   0          46h
csi-s3-kzxkn             2/2     Running   0          46h
csi-s3-ltjx8             2/2     Running   0          46h
csi-s3-nwt4w             2/2     Running   0          46h
csi-s3-vdlps             2/2     Running   0          46h
csi-s3-x25ng             2/2     Running   0          46h
kafka-cluster-broker-0   2/2     Running   0          3m41s

k get ops
NAME                                TYPE            CLUSTER         STATUS     PROGRESS   AGE
kafka-cluster-reconfiguring-wvdjn   Reconfiguring   kafka-cluster   Creating   -/-        4m
  1. see logs
k describe cluster kafka-cluster
Name:         kafka-cluster
Namespace:    default
Labels:       app.kubernetes.io/instance=kafka-cluster
              app.kubernetes.io/version=3.3.2
              clusterdefinition.kubeblocks.io/name=kafka
              clusterversion.kubeblocks.io/name=kafka-3.3.2
              helm.sh/chart=kafka-cluster-0.8.0-alpha.0
Annotations:  kubeblocks.io/extra-env:
                {"KB_KAFKA_ENABLE_SASL":"false","KB_KAFKA_BROKER_HEAP":"-XshowSettings:vm -XX:MaxRAMPercentage=100 -Ddepth=64","KB_KAFKA_CONTROLLER_HEAP":...
              kubeblocks.io/ops-request: [{"name":"kafka-cluster-reconfiguring-wvdjn","type":"Reconfiguring"}]
              kubeblocks.io/reconcile: 2023-10-25T06:02:03.120378032Z
API Version:  apps.kubeblocks.io/v1alpha1
Kind:         Cluster
Metadata:
  Creation Timestamp:  2023-10-25T06:00:19Z
  Finalizers:
    cluster.kubeblocks.io/finalizer
  Generation:        2
  Resource Version:  5137518
  UID:               39eb973e-ece4-4ac3-bb2e-26c65f64a033
Spec:
  Affinity:
    Pod Anti Affinity:  Preferred
    Tenancy:            SharedNode
    Topology Keys:
      kubernetes.io/hostname
  Cluster Definition Ref:  kafka
  Cluster Version Ref:     kafka-3.3.2
  Component Specs:
    Component Def Ref:  kafka-server
    Monitor:            false
    Name:               broker
    No Create PDB:      false
    Replicas:           1
    Resources:
      Limits:
        Cpu:     500m
        Memory:  512Mi
      Requests:
        Cpu:               500m
        Memory:            512Mi
    Service Account Name:  kb-kafka-cluster
  Monitor:
  Resources:
    Cpu:     0
    Memory:  0
  Storage:
    Size:              0
  Termination Policy:  Delete
Status:
  Cluster Def Generation:  2
  Components:
    Broker:
      Phase:            Running
      Pods Ready:       true
      Pods Ready Time:  2023-10-25T06:00:50Z
  Conditions:
    Last Transition Time:  2023-10-25T06:00:19Z
    Message:               The operator has started the provisioning of Cluster: kafka-cluster
    Observed Generation:   1
    Reason:                PreCheckSucceed
    Status:                True
    Type:                  ProvisioningStarted
    Last Transition Time:  2023-10-25T06:00:50Z
    Message:               Successfully applied for resources
    Observed Generation:   1
    Reason:                ApplyResourcesSucceed
    Status:                True
    Type:                  ApplyResources
    Last Transition Time:  2023-10-25T06:00:50Z
    Message:               all pods of components are ready, waiting for the probe detection successful
    Reason:                AllReplicasReady
    Status:                True
    Type:                  ReplicasReady
    Last Transition Time:  2023-10-25T06:00:50Z
    Message:               Cluster: kafka-cluster is ready, current phase is Running
    Reason:                ClusterReady
    Status:                True
    Type:                  Ready
  Observed Generation:     1
  Phase:                   Running
Events:
  Type     Reason                    Age                    From                Message
  ----     ------                    ----                   ----                -------
  Normal   PreCheckSucceed           4m15s                  cluster-controller  The operator has started the provisioning of Cluster: kafka-cluster
  Normal   ComponentPhaseTransition  4m14s (x2 over 4m15s)  cluster-controller  Create a new component
  Normal   ApplyResourcesSucceed     3m45s (x3 over 4m15s)  cluster-controller  Successfully applied for resources
  Warning  ApplyResourcesFailed      3m45s (x2 over 4m15s)  cluster-controller  Operation cannot be fulfilled on replicatedstatemachines.workloads.kubeblocks.io "kafka-cluster-broker": the object has been modified; please apply your changes to the latest version and try again
  Normal   ComponentPhaseTransition  3m45s                  cluster-controller  component is Running
  Normal   AllReplicasReady          3m45s                  cluster-controller  all pods of components are ready, waiting for the probe detection successful
  Normal   ClusterReady              3m45s                  cluster-controller  Cluster: kafka-cluster is ready, current phase is Running
  Normal   Running                   3m45s                  cluster-controller  Cluster: kafka-cluster is ready, current phase is Running
  Warning  FailedMount               2m32s                  event-controller    Pod kafka-cluster-broker-0: MountVolume.SetUp failed for volume "kube-api-access-htq6n" : failed to fetch token: serviceaccounts "kb-kafka-cluster" not found

k describe ops kafka-cluster-reconfiguring-wvdjn
Name:         kafka-cluster-reconfiguring-wvdjn
Namespace:    default
Labels:       app.kubernetes.io/instance=kafka-cluster
              ops.kubeblocks.io/ops-type=Reconfiguring
Annotations:  <none>
API Version:  apps.kubeblocks.io/v1alpha1
Kind:         OpsRequest
Metadata:
  Creation Timestamp:  2023-10-25T06:01:07Z
  Finalizers:
    opsrequest.kubeblocks.io/finalizer
  Generate Name:  kafka-cluster-reconfiguring-
  Generation:     1
  Owner References:
    API Version:     apps.kubeblocks.io/v1alpha1
    Kind:            Cluster
    Name:            kafka-cluster
    UID:             39eb973e-ece4-4ac3-bb2e-26c65f64a033
  Resource Version:  5137245
  UID:               cd5857d8-f873-4073-a58c-867ededd97e0
Spec:
  Cluster Ref:  kafka-cluster
  Reconfigure:
    Component Name:  broker
    Configurations:
      Keys:
        Key:  kafka-configuration-tpl
        Parameters:
          Key:               MaxRAMPercentage
          Value:             200
          Key:               MaxGCPauseMillis
          Value:             30
      Name:                  mysql-consensusset-config
  Ttl Seconds Before Abort:  0
  Type:                      Reconfiguring
Status:
  Conditions:
    Last Transition Time:  2023-10-25T06:01:07Z
    Message:               Start to process the OpsRequest: kafka-cluster-reconfiguring-wvdjn in Cluster: kafka-cluster
    Reason:                OpsRequestProgressingStarted
    Status:                True
    Type:                  Progressing
    Last Transition Time:  2023-10-25T06:01:07Z
    Message:               OpsRequest: kafka-cluster-reconfiguring-wvdjn is validated
    Reason:                ValidateOpsRequestPassed
    Status:                True
    Type:                  Validated
    Last Transition Time:  2023-10-25T06:01:07Z
    Message:               Start to reconfigure in Cluster: kafka-cluster, Component: broker
    Reason:                ReconfigureStarted
    Status:                True
    Type:                  Reconfigure
  Phase:                   Creating
  Progress:                -/-
  Start Timestamp:         2023-10-25T06:01:07Z
Events:
  Type    Reason                        Age    From                    Message
  ----    ------                        ----   ----                    -------
  Normal  OpsRequestProgressingStarted  4m23s  ops-request-controller  Start to process the OpsRequest: kafka-cluster-reconfiguring-wvdjn in Cluster: kafka-cluster
  Normal  ValidateOpsRequestPassed      4m23s  ops-request-controller  OpsRequest: kafka-cluster-reconfiguring-wvdjn is validated
  Normal  ReconfigureStarted            4m23s  ops-request-controller  Start to reconfigure in Cluster: kafka-cluster, Component: broker

k get pod -n kb-system
NAME                                            READY   STATUS             RESTARTS      AGE
kb-addon-snapshot-controller-6fb4fb9b55-4g7v4   1/1     Running            0             8m25s
kubeblocks-7f6b8557bb-dp9bb                     0/1     CrashLoopBackOff   5 (15s ago)   8m38s
kubeblocks-dataprotection-6bd57bc9d4-5mzlq      1/1     Running            0             8m38s

k logs kubeblocks-7f6b8557bb-dp9bb -n kb-system
Defaulted container "manager" out of: manager, tools (init), datascript (init)
2023-10-25T06:05:18.083Z	INFO	setup	config file: /etc/kubeblocks/config.yaml
2023-10-25T06:05:18.084Z	INFO	setup	config settings: map[apps:true backup_pv_configmap_name: backup_pv_configmap_namespace: backup_pvc_create_policy: backup_pvc_init_capacity: backup_pvc_name: backup_pvc_storage_class: cert_dir:/tmp/k8s-webhook-server/serving-certs cm_namespace:kb-system cm_recon_retry_duration_ms:1000 config_manager_grpc_port:9901 config_manager_log_level:info data_plane_affinity:{"nodeAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"preference":{"matchExpressions":[{"key":"kb-data","operator":"In","values":["true"]}]},"weight":100}]}} data_plane_tolerations:[{"effect":"NoSchedule","key":"kb-data","operator":"Equal","value":"true"}] default_storage_class:kb-default-sc enable_debug_sysaccounts:false enablerbacmanager:true extensions:true health_probe_bind_address::8081 kubeblocks_addon_helm_install_options:--atomic --cleanup-on-fail --wait --insecure-skip-tls-verify kubeblocks_addon_helm_uninstall_options:[] kubeblocks_addon_sa_name:kubeblocks-addon-installer kubeblocks_datascript_clients_image:infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/kubeblocks-datascript:0.7.0-beta.10 kubeblocks_serviceaccount_name:kubeblocks kubeblocks_tools_image:infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/kubeblocks-tools:0.7.0-beta.10 kubeconfig: kubernetes_cluster_domain:cluster.local leader_elect:true leader_elect_id:001c317f maxconcurrentreconciles_addon:4 maxconcurrentreconciles_clusterdef:2 maxconcurrentreconciles_clusterversion:2 metrics_bind_address::8080 pod_min_ready_seconds:10 probe_service_grpc_port:50001 probe_service_http_port:3501 probe_service_log_level:info replicated_state_machine:true storage:true volumesnapshot_api_beta:false workloads:true zap_devel:false zap_encoder:console zap_log_level: zap_stacktrace_level: zap_time_encoding:iso8601]
2023-10-25T06:05:18.085Z	INFO	controller-runtime.metrics	Metrics server is starting to listen	{"addr": ":8080"}
2023-10-25T06:05:18.093Z	INFO	setup	starting manager
2023-10-25T06:05:18.094Z	INFO	Starting server	{"kind": "health probe", "addr": "[::]:8081"}
2023-10-25T06:05:18.094Z	INFO	starting server	{"path": "/metrics", "kind": "metrics", "addr": "[::]:8080"}
I1025 06:05:18.094348       1 leaderelection.go:250] attempting to acquire leader lease kb-system/001c317f.kubeblocks.io...
I1025 06:05:34.535282       1 leaderelection.go:260] successfully acquired lease kb-system/001c317f.kubeblocks.io
2023-10-25T06:05:34.535Z	INFO	Starting EventSource	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1alpha1.Cluster"}
2023-10-25T06:05:34.535Z	INFO	Starting EventSource	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1.StatefulSet"}
2023-10-25T06:05:34.535Z	INFO	Starting EventSource	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1.Deployment"}
2023-10-25T06:05:34.535Z	INFO	Starting EventSource	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1alpha1.ReplicatedStateMachine"}
2023-10-25T06:05:34.535Z	INFO	Starting EventSource	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1.Service"}
2023-10-25T06:05:34.535Z	INFO	Starting EventSource	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1.Secret"}
2023-10-25T06:05:34.535Z	INFO	Starting EventSource	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1.ConfigMap"}
2023-10-25T06:05:34.535Z	INFO	Starting EventSource	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1.PersistentVolumeClaim"}
2023-10-25T06:05:34.535Z	INFO	Starting EventSource	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1.PodDisruptionBudget"}
2023-10-25T06:05:34.535Z	INFO	Starting EventSource	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1alpha1.BackupPolicy"}
2023-10-25T06:05:34.535Z	INFO	Starting EventSource	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1alpha1.BackupSchedule"}
2023-10-25T06:05:34.535Z	INFO	Starting EventSource	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1alpha1.Backup"}
2023-10-25T06:05:34.535Z	INFO	Starting EventSource	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1alpha1.Restore"}
2023-10-25T06:05:34.535Z	INFO	Starting EventSource	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1.Job"}
2023-10-25T06:05:34.535Z	INFO	Starting EventSource	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1.ClusterRoleBinding"}
2023-10-25T06:05:34.535Z	INFO	Starting EventSource	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1.RoleBinding"}
2023-10-25T06:05:34.535Z	INFO	Starting EventSource	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1.ServiceAccount"}
2023-10-25T06:05:34.535Z	INFO	Starting EventSource	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1.Pod"}
2023-10-25T06:05:34.535Z	INFO	Starting Controller	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster"}
2023-10-25T06:05:34.535Z	INFO	Starting EventSource	{"controller": "clusterversion", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ClusterVersion", "source": "kind source: *v1alpha1.ClusterVersion"}
2023-10-25T06:05:34.536Z	INFO	Starting Controller	{"controller": "clusterversion", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ClusterVersion"}
2023-10-25T06:05:34.536Z	INFO	Starting EventSource	{"controller": "clusterdefinition", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ClusterDefinition", "source": "kind source: *v1alpha1.ClusterDefinition"}
2023-10-25T06:05:34.536Z	INFO	Starting Controller	{"controller": "clusterdefinition", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ClusterDefinition"}
2023-10-25T06:05:34.536Z	INFO	Starting EventSource	{"controller": "configconstraint", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ConfigConstraint", "source": "kind source: *v1alpha1.ConfigConstraint"}
2023-10-25T06:05:34.536Z	INFO	Starting EventSource	{"controller": "configconstraint", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ConfigConstraint", "source": "kind source: *v1.ConfigMap"}
2023-10-25T06:05:34.536Z	INFO	Starting Controller	{"controller": "configconstraint", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ConfigConstraint"}
2023-10-25T06:05:34.536Z	INFO	Starting EventSource	{"controller": "configuration", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Configuration", "source": "kind source: *v1alpha1.Configuration"}
2023-10-25T06:05:34.536Z	INFO	Starting EventSource	{"controller": "configuration", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Configuration", "source": "kind source: *v1.ConfigMap"}
2023-10-25T06:05:34.536Z	INFO	Starting Controller	{"controller": "configuration", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Configuration"}
2023-10-25T06:05:34.536Z	INFO	Starting EventSource	{"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "source": "kind source: *v1alpha1.OpsRequest"}
2023-10-25T06:05:34.536Z	INFO	Starting EventSource	{"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "source": "kind source: *v1alpha1.Cluster"}
2023-10-25T06:05:34.536Z	INFO	Starting EventSource	{"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "source": "kind source: *v1alpha1.Backup"}
2023-10-25T06:05:34.536Z	INFO	Starting EventSource	{"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "source": "kind source: *v1.PersistentVolumeClaim"}
2023-10-25T06:05:34.536Z	INFO	Starting EventSource	{"controller": "event", "controllerGroup": "", "controllerKind": "Event", "source": "kind source: *v1.Event"}
2023-10-25T06:05:34.536Z	INFO	Starting Controller	{"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest"}
2023-10-25T06:05:34.536Z	INFO	Starting Controller	{"controller": "event", "controllerGroup": "", "controllerKind": "Event"}
2023-10-25T06:05:34.536Z	INFO	Starting EventSource	{"controller": "configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "source": "kind source: *v1.ConfigMap"}
2023-10-25T06:05:34.536Z	INFO	Starting Controller	{"controller": "configmap", "controllerGroup": "", "controllerKind": "ConfigMap"}
2023-10-25T06:05:34.536Z	INFO	Starting EventSource	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1alpha1.Cluster"}
2023-10-25T06:05:34.536Z	INFO	Starting EventSource	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1.Secret"}
2023-10-25T06:05:34.536Z	INFO	Starting EventSource	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1.Job"}
2023-10-25T06:05:34.536Z	INFO	Starting Controller	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster"}
2023-10-25T06:05:34.536Z	INFO	Starting EventSource	{"controller": "servicedescriptor", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ServiceDescriptor", "source": "kind source: *v1alpha1.ServiceDescriptor"}
2023-10-25T06:05:34.536Z	INFO	Starting Controller	{"controller": "servicedescriptor", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ServiceDescriptor"}
2023-10-25T06:05:34.536Z	INFO	Starting EventSource	{"controller": "componentclassdefinition", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ComponentClassDefinition", "source": "kind source: *v1alpha1.ComponentClassDefinition"}
2023-10-25T06:05:34.536Z	INFO	Starting EventSource	{"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "source": "kind source: *v1alpha1.ReplicatedStateMachine"}
2023-10-25T06:05:34.536Z	INFO	Starting Controller	{"controller": "componentclassdefinition", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ComponentClassDefinition"}
2023-10-25T06:05:34.536Z	INFO	Starting EventSource	{"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "source": "kind source: *v1.StatefulSet"}
2023-10-25T06:05:34.537Z	INFO	Starting EventSource	{"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "source": "kind source: *v1.Job"}
2023-10-25T06:05:34.537Z	INFO	Starting EventSource	{"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "source": "kind source: *v1.Pod"}
2023-10-25T06:05:34.537Z	INFO	Starting Controller	{"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine"}
2023-10-25T06:05:34.537Z	INFO	Starting EventSource	{"controller": "storageprovider", "controllerGroup": "storage.kubeblocks.io", "controllerKind": "StorageProvider", "source": "kind source: *v1alpha1.StorageProvider"}
2023-10-25T06:05:34.537Z	INFO	Starting EventSource	{"controller": "storageprovider", "controllerGroup": "storage.kubeblocks.io", "controllerKind": "StorageProvider", "source": "kind source: *v1.CSIDriver"}
2023-10-25T06:05:34.537Z	INFO	Starting Controller	{"controller": "storageprovider", "controllerGroup": "storage.kubeblocks.io", "controllerKind": "StorageProvider"}
2023-10-25T06:05:34.537Z	INFO	Starting EventSource	{"controller": "addon", "controllerGroup": "extensions.kubeblocks.io", "controllerKind": "Addon", "source": "kind source: *v1alpha1.Addon"}
2023-10-25T06:05:34.537Z	INFO	Starting EventSource	{"controller": "addon", "controllerGroup": "extensions.kubeblocks.io", "controllerKind": "Addon", "source": "kind source: *v1.Job"}
2023-10-25T06:05:34.537Z	INFO	Starting Controller	{"controller": "addon", "controllerGroup": "extensions.kubeblocks.io", "controllerKind": "Addon"}
2023-10-25T06:05:34.754Z	INFO	Starting workers	{"controller": "event", "controllerGroup": "", "controllerKind": "Event", "worker count": 1}
2023-10-25T06:05:34.761Z	INFO	Starting workers	{"controller": "configuration", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Configuration", "worker count": 1}
2023-10-25T06:05:34.775Z	INFO	Starting workers	{"controller": "servicedescriptor", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ServiceDescriptor", "worker count": 1}
2023-10-25T06:05:34.789Z	INFO	Starting workers	{"controller": "configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "worker count": 1}
2023-10-25T06:05:34.793Z	INFO	Starting workers	{"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "worker count": 1}
2023-10-25T06:05:34.795Z	INFO	Starting workers	{"controller": "addon", "controllerGroup": "extensions.kubeblocks.io", "controllerKind": "Addon", "worker count": 4}
2023-10-25T06:05:34.796Z	INFO	Starting workers	{"controller": "clusterversion", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ClusterVersion", "worker count": 2}
2023-10-25T06:05:34.805Z	INFO	DAG: |->{obj:*v1.Service, name: kafka-cluster-broker-headless, action: UPDATE}->{obj:*v1.ConfigMap, name: kafka-cluster-broker-rsm-env, action: UPDATE}->{obj:*v1.Service, name: kafka-cluster-broker, action: UPDATE}->{obj:*v1.StatefulSet, name: kafka-cluster-broker, action: UPDATE}->{obj:*v1alpha1.ReplicatedStateMachine, name: kafka-cluster-broker, action: STATUS}	{"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"kafka-cluster-broker","namespace":"default"}, "namespace": "default", "name": "kafka-cluster-broker", "reconcileID": "b6a49adc-b5af-4efa-9af2-cac2a25971df", "ReplicatedStateMachine": {"name":"kafka-cluster-broker","namespace":"default"}}
2023-10-25T06:05:34.819Z	INFO	Starting workers	{"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "worker count": 1}
2023-10-25T06:05:34.820Z	INFO	Starting workers	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "worker count": 1}
2023-10-25T06:05:34.820Z	INFO	Starting workers	{"controller": "componentclassdefinition", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ComponentClassDefinition", "worker count": 1}
2023-10-25T06:05:34.821Z	INFO	Starting workers	{"controller": "clusterdefinition", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ClusterDefinition", "worker count": 2}
2023-10-25T06:05:34.821Z	INFO	Starting workers	{"controller": "configconstraint", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ConfigConstraint", "worker count": 1}
2023-10-25T06:05:34.835Z	INFO	Starting workers	{"controller": "storageprovider", "controllerGroup": "storage.kubeblocks.io", "controllerKind": "StorageProvider", "worker count": 1}
2023-10-25T06:05:34.851Z	INFO	Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference	{"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "OpsRequest": {"name":"kafka-cluster-reconfiguring-wvdjn","namespace":"default"}, "namespace": "default", "name": "kafka-cluster-reconfiguring-wvdjn", "reconcileID": "10da98d5-67dc-48ff-8fc9-abb3dafd690b"}
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x68 pc=0x1c58dcd]

goroutine 684 [running]:
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:115 +0x1e5
panic({0x1f28d40?, 0x390cdc0?})
	/usr/local/go/src/runtime/panic.go:914 +0x21f
github.com/apecloud/kubeblocks/controllers/apps/operations.(*pipeline).ConfigConstraints(0xc002a3e120?)
	/src/controllers/apps/operations/pipeline.go:131 +0x2d
github.com/apecloud/kubeblocks/controllers/apps/operations.(*reconfigureAction).Action(0xc001f53020?, {{0x2742610, 0xc001f52e70}, {{{0xc0014af056, 0x7}, {0xc0014c60f0, 0x21}}}, {{0x2747578, 0xc001f52ed0}, 0x0}, ...}, ...)
	/src/controllers/apps/operations/reconfigure.go:287 +0x3de
github.com/apecloud/kubeblocks/controllers/apps/operations.(*OpsManager).Do(0x410405?, {{0x2742610, 0xc001f52e70}, {{{0xc0014af056, 0x7}, {0xc0014c60f0, 0x21}}}, {{0x2747578, 0xc001f52ed0}, 0x0}, ...}, ...)
	/src/controllers/apps/operations/ops_manager.go:96 +0x50d
github.com/apecloud/kubeblocks/controllers/apps.(*OpsRequestReconciler).doOpsRequestAction(0xc00057b3b0, {{0x2742610, 0xc001f52e70}, {{{0xc0014af056, 0x7}, {0xc0014c60f0, 0x21}}}, {{0x2747578, 0xc001f52ed0}, 0x0}, ...}, ...)
	/src/controllers/apps/opsrequest_controller.go:247 +0xc5
github.com/apecloud/kubeblocks/controllers/apps.(*OpsRequestReconciler).handleOpsRequestByPhase(0xc00057b3b0?, {{0x2742610, 0xc001f52e70}, {{{0xc0014af056, 0x7}, {0xc0014c60f0, 0x21}}}, {{0x2747578, 0xc001f52ed0}, 0x0}, ...}, ...)
	/src/controllers/apps/opsrequest_controller.go:148 +0x298
github.com/apecloud/kubeblocks/controllers/apps.(*opsControllerHandler).Handle(0x7fcf9f40f5b8?, {{0x2742610, 0xc001f52e70}, {{{0xc0014af056, 0x7}, {0xc0014c60f0, 0x21}}}, {{0x2747578, 0xc001f52ed0}, 0x0}, ...}, ...)
	/src/controllers/apps/opsrequest_controller.go:361 +0x99
github.com/apecloud/kubeblocks/controllers/apps.(*OpsRequestReconciler).Reconcile(0xc00057b3b0, {0x2742610, 0xc001f52e70}, {{{0xc0014af056, 0x7}, {0xc0014c60f0, 0x21}}})
	/src/controllers/apps/opsrequest_controller.go:72 +0x3f3
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x2742610?, {0x2742610?, 0xc001f52e70?}, {{{0xc0014af056?, 0x1ded780?}, {0xc0014c60f0?, 0x2732140?}}})
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:118 +0xb7
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000673900, {0x2742648, 0xc000192910}, {0x200cc20?, 0xc0006749a0?})
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:314 +0x365
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000673900, {0x2742648, 0xc000192910})
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:265 +0x1c9
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:226 +0x79
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 76
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222 +0x565
@linghan-hub linghan-hub added kind/bug Something isn't working severity/major Great chance user will encounter the same problem labels Oct 25, 2023
@linghan-hub linghan-hub added this to the Release 0.7.0 milestone Oct 25, 2023
@linghan-hub linghan-hub changed the title [BUG] create mysql cluster lead to kubeblocks CrashLoopBackOff [BUG] kafka cluster execute ops Reconfiguring lead to kubeblocks CrashLoopBackOff Oct 25, 2023
@linghan-hub linghan-hub assigned sophon-zt and unassigned free6om Oct 25, 2023
@sophon-zt sophon-zt linked a pull request Oct 25, 2023 that will close this issue
fix: incorrect state for reconfiguring operator #5570
Merged
@sophon-zt
Copy link
Contributor

This bug has been fixed in this PR: #5570

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sophon-zt sophon-zt

Labels
bug kind/bug Something isn't working severity/major Great chance user will encounter the same problem
Projects
None yet
Milestone
Release 0.7.0
Development

Successfully merging a pull request may close this issue.

fix: incorrect state for reconfiguring operator apecloud/kubeblocks
4 participants
@free6om @linghan-hub @sophon-zt @apecloud-bot