Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] kubeblocks pod CrashLoopBackOff ReplicatedStateMachine Reconcile panic #5504

Closed
JashBook opened this issue Oct 18, 2023 · 0 comments · Fixed by #5518
Closed

[BUG] kubeblocks pod CrashLoopBackOff ReplicatedStateMachine Reconcile panic #5504

JashBook opened this issue Oct 18, 2023 · 0 comments · Fixed by #5518
Assignees
@free6om
Labels
bug ci kind/bug Something isn't working severity/critical Blocking or critical issues
Milestone
Release 0.7.0

Comments

@JashBook
Copy link
Collaborator

JashBook commented Oct 18, 2023 •
edited
Loading

Describe the bug
kubeblocks pod CrashLoopBackOff ReplicatedStateMachine Reconcile panic

kbcli version
Kubernetes: v1.26.7-gke.500
KubeBlocks: 0.7.0-beta.6
kbcli: 0.7.0-beta.6

To Reproduce
Steps to reproduce the behavior:

  1. create cluster starrock
  2. See error
kubectl logs kubeblocks-8556684979-2zwgl -n kb-irkofnhzoj
Defaulted container "manager" out of: manager, tools (init), datascript (init)
2023-10-18T03:28:51.287Z	INFO	setup	config file: /etc/kubeblocks/config.yaml
2023-10-18T03:28:51.287Z	INFO	setup	config settings: map[apps:true backup_pv_configmap_name: backup_pv_configmap_namespace: backup_pvc_create_policy: backup_pvc_init_capacity: backup_pvc_name: backup_pvc_storage_class: cert_dir:/tmp/k8s-webhook-server/serving-certs cm_namespace:kb-irkofnhzoj cm_recon_retry_duration_ms:1000 config_manager_grpc_port:9901 config_manager_log_level:info data_plane_affinity:{"nodeAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"preference":{"matchExpressions":[{"key":"kb-data","operator":"In","values":["true"]}]},"weight":100}]}} data_plane_tolerations:[{"effect":"NoSchedule","key":"kb-data","operator":"Equal","value":"true"}] default_storage_class:kb-default-sc enable_debug_sysaccounts:false enablerbacmanager:true extensions:true health_probe_bind_address::8081 kubeblocks_addon_helm_install_options:--atomic --cleanup-on-fail --wait --insecure-skip-tls-verify kubeblocks_addon_helm_uninstall_options:[] kubeblocks_addon_sa_name:kubeblocks-addon-installer kubeblocks_datascript_clients_image:docker.io/apecloud/kubeblocks-datascript:0.7.0-beta.6 kubeblocks_serviceaccount_name:kubeblocks kubeblocks_tools_image:docker.io/apecloud/kubeblocks-tools:0.7.0-beta.6 kubeconfig: kubernetes_cluster_domain:cluster.local leader_elect:true leader_elect_id:001c317f maxconcurrentreconciles_addon:8 maxconcurrentreconciles_clusterdef:4 maxconcurrentreconciles_clusterversion:4 metrics_bind_address::8080 pod_min_ready_seconds:10 probe_service_grpc_port:50001 probe_service_http_port:3501 probe_service_log_level:info replicated_state_machine:true storage:true volumesnapshot_api_beta:false workloads:true zap_devel:false zap_encoder:console zap_log_level: zap_stacktrace_level: zap_time_encoding:iso8601]
2023-10-18T03:28:51.288Z	INFO	controller-runtime.metrics	Metrics server is starting to listen	{"addr": ":8080"}
2023-10-18T03:28:51.303Z	INFO	setup	starting manager
2023-10-18T03:28:51.303Z	INFO	Starting server	{"kind": "health probe", "addr": "[::]:8081"}
2023-10-18T03:28:51.304Z	INFO	starting server	{"path": "/metrics", "kind": "metrics", "addr": "[::]:8080"}
I1018 03:28:51.304256       1 leaderelection.go:250] attempting to acquire leader lease kb-irkofnhzoj/001c317f.kubeblocks.io...
I1018 03:29:07.178435       1 leaderelection.go:260] successfully acquired lease kb-irkofnhzoj/001c317f.kubeblocks.io
2023-10-18T03:29:07.178Z	INFO	Starting EventSource	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1alpha1.Cluster"}
2023-10-18T03:29:07.178Z	INFO	Starting EventSource	{"controller": "clusterversion", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ClusterVersion", "source": "kind source: *v1alpha1.ClusterVersion"}
2023-10-18T03:29:07.178Z	INFO	Starting Controller	{"controller": "clusterversion", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ClusterVersion"}
2023-10-18T03:29:07.179Z	INFO	Starting EventSource	{"controller": "clusterdefinition", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ClusterDefinition", "source": "kind source: *v1alpha1.ClusterDefinition"}
2023-10-18T03:29:07.179Z	INFO	Starting Controller	{"controller": "clusterdefinition", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ClusterDefinition"}
2023-10-18T03:29:07.179Z	INFO	Starting EventSource	{"controller": "configconstraint", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ConfigConstraint", "source": "kind source: *v1alpha1.ConfigConstraint"}
2023-10-18T03:29:07.179Z	INFO	Starting EventSource	{"controller": "configconstraint", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ConfigConstraint", "source": "kind source: *v1.ConfigMap"}
2023-10-18T03:29:07.179Z	INFO	Starting Controller	{"controller": "configconstraint", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ConfigConstraint"}
2023-10-18T03:29:07.179Z	INFO	Starting EventSource	{"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "source": "kind source: *v1alpha1.OpsRequest"}
2023-10-18T03:29:07.179Z	INFO	Starting EventSource	{"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "source": "kind source: *v1alpha1.Cluster"}
2023-10-18T03:29:07.179Z	INFO	Starting EventSource	{"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "source": "kind source: *v1alpha1.Backup"}
2023-10-18T03:29:07.179Z	INFO	Starting EventSource	{"controller": "configuration", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Configuration", "source": "kind source: *v1alpha1.Configuration"}
2023-10-18T03:29:07.179Z	INFO	Starting EventSource	{"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "source": "kind source: *v1.PersistentVolumeClaim"}
2023-10-18T03:29:07.179Z	INFO	Starting Controller	{"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest"}
2023-10-18T03:29:07.179Z	INFO	Starting EventSource	{"controller": "configuration", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Configuration", "source": "kind source: *v1.ConfigMap"}
2023-10-18T03:29:07.179Z	INFO	Starting Controller	{"controller": "configuration", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Configuration"}
2023-10-18T03:29:07.179Z	INFO	Starting EventSource	{"controller": "configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "source": "kind source: *v1.ConfigMap"}
2023-10-18T03:29:07.179Z	INFO	Starting Controller	{"controller": "configmap", "controllerGroup": "", "controllerKind": "ConfigMap"}
2023-10-18T03:29:07.179Z	INFO	Starting EventSource	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1.StatefulSet"}
2023-10-18T03:29:07.179Z	INFO	Starting EventSource	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1alpha1.Cluster"}
2023-10-18T03:29:07.179Z	INFO	Starting EventSource	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1.Deployment"}
2023-10-18T03:29:07.179Z	INFO	Starting EventSource	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1.Secret"}
2023-10-18T03:29:07.179Z	INFO	Starting EventSource	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1alpha1.ReplicatedStateMachine"}
2023-10-18T03:29:07.179Z	INFO	Starting EventSource	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1.Job"}
2023-10-18T03:29:07.179Z	INFO	Starting Controller	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster"}
2023-10-18T03:29:07.179Z	INFO	Starting EventSource	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1.Service"}
2023-10-18T03:29:07.179Z	INFO	Starting EventSource	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1.Secret"}
2023-10-18T03:29:07.179Z	INFO	Starting EventSource	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1.ConfigMap"}
2023-10-18T03:29:07.179Z	INFO	Starting EventSource	{"controller": "event", "controllerGroup": "", "controllerKind": "Event", "source": "kind source: *v1.Event"}
2023-10-18T03:29:07.179Z	INFO	Starting Controller	{"controller": "event", "controllerGroup": "", "controllerKind": "Event"}
2023-10-18T03:29:07.179Z	INFO	Starting EventSource	{"controller": "componentclassdefinition", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ComponentClassDefinition", "source": "kind source: *v1alpha1.ComponentClassDefinition"}
2023-10-18T03:29:07.179Z	INFO	Starting Controller	{"controller": "componentclassdefinition", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ComponentClassDefinition"}
2023-10-18T03:29:07.179Z	INFO	Starting EventSource	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1.PersistentVolumeClaim"}
2023-10-18T03:29:07.179Z	INFO	Starting EventSource	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1.PodDisruptionBudget"}
2023-10-18T03:29:07.179Z	INFO	Starting EventSource	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1alpha1.BackupPolicy"}
2023-10-18T03:29:07.179Z	INFO	Starting EventSource	{"controller": "servicedescriptor", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ServiceDescriptor", "source": "kind source: *v1alpha1.ServiceDescriptor"}
2023-10-18T03:29:07.179Z	INFO	Starting Controller	{"controller": "servicedescriptor", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ServiceDescriptor"}
2023-10-18T03:29:07.179Z	INFO	Starting EventSource	{"controller": "storageprovider", "controllerGroup": "storage.kubeblocks.io", "controllerKind": "StorageProvider", "source": "kind source: *v1alpha1.StorageProvider"}
2023-10-18T03:29:07.179Z	INFO	Starting EventSource	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1alpha1.BackupSchedule"}
2023-10-18T03:29:07.179Z	INFO	Starting EventSource	{"controller": "storageprovider", "controllerGroup": "storage.kubeblocks.io", "controllerKind": "StorageProvider", "source": "kind source: *v1.CSIDriver"}
2023-10-18T03:29:07.179Z	INFO	Starting EventSource	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1alpha1.Backup"}
2023-10-18T03:29:07.179Z	INFO	Starting EventSource	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1alpha1.Restore"}
2023-10-18T03:29:07.179Z	INFO	Starting EventSource	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1.Job"}
2023-10-18T03:29:07.179Z	INFO	Starting EventSource	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1.ClusterRoleBinding"}
2023-10-18T03:29:07.179Z	INFO	Starting Controller	{"controller": "storageprovider", "controllerGroup": "storage.kubeblocks.io", "controllerKind": "StorageProvider"}
2023-10-18T03:29:07.179Z	INFO	Starting EventSource	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1.RoleBinding"}
2023-10-18T03:29:07.179Z	INFO	Starting EventSource	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1.ServiceAccount"}
2023-10-18T03:29:07.179Z	INFO	Starting EventSource	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1.Pod"}
2023-10-18T03:29:07.179Z	INFO	Starting EventSource	{"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "source": "kind source: *v1alpha1.ReplicatedStateMachine"}
2023-10-18T03:29:07.179Z	INFO	Starting EventSource	{"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "source": "kind source: *v1.StatefulSet"}
2023-10-18T03:29:07.179Z	INFO	Starting EventSource	{"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "source": "kind source: *v1.Job"}
2023-10-18T03:29:07.179Z	INFO	Starting EventSource	{"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "source": "kind source: *v1.Pod"}
2023-10-18T03:29:07.179Z	INFO	Starting Controller	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster"}
2023-10-18T03:29:07.179Z	INFO	Starting Controller	{"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine"}
2023-10-18T03:29:07.179Z	INFO	Starting EventSource	{"controller": "addon", "controllerGroup": "extensions.kubeblocks.io", "controllerKind": "Addon", "source": "kind source: *v1alpha1.Addon"}
2023-10-18T03:29:07.179Z	INFO	Starting EventSource	{"controller": "addon", "controllerGroup": "extensions.kubeblocks.io", "controllerKind": "Addon", "source": "kind source: *v1.Job"}
2023-10-18T03:29:07.179Z	INFO	Starting Controller	{"controller": "addon", "controllerGroup": "extensions.kubeblocks.io", "controllerKind": "Addon"}
2023-10-18T03:29:07.842Z	INFO	Starting workers	{"controller": "addon", "controllerGroup": "extensions.kubeblocks.io", "controllerKind": "Addon", "worker count": 8}
2023-10-18T03:29:07.849Z	INFO	Starting workers	{"controller": "clusterdefinition", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ClusterDefinition", "worker count": 4}
2023-10-18T03:29:07.877Z	INFO	Starting workers	{"controller": "configconstraint", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ConfigConstraint", "worker count": 1}
2023-10-18T03:29:07.878Z	INFO	Starting workers	{"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "worker count": 1}
2023-10-18T03:29:07.878Z	INFO	Starting workers	{"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "worker count": 1}
2023-10-18T03:29:07.886Z	INFO	Starting workers	{"controller": "event", "controllerGroup": "", "controllerKind": "Event", "worker count": 1}
2023-10-18T03:29:07.886Z	INFO	Starting workers	{"controller": "componentclassdefinition", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ComponentClassDefinition", "worker count": 1}
2023-10-18T03:29:07.886Z	INFO	Starting workers	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "worker count": 1}
2023-10-18T03:29:07.886Z	INFO	Starting workers	{"controller": "configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "worker count": 1}
2023-10-18T03:29:07.886Z	INFO	Starting workers	{"controller": "configuration", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Configuration", "worker count": 1}
2023-10-18T03:29:07.887Z	INFO	Starting workers	{"controller": "clusterversion", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ClusterVersion", "worker count": 4}
2023-10-18T03:29:07.900Z	INFO	action list: []
	{"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"smarte-bxggqi-mysql","namespace":"ns-zpnclfdcyx"}, "namespace": "ns-zpnclfdcyx", "name": "smarte-bxggqi-mysql", "reconcileID": "0b2a9b64-37f9-4a76-92b7-07cf31d015ee", "ReplicatedStateMachine": {"name":"smarte-bxggqi-mysql","namespace":"ns-zpnclfdcyx"}}
2023-10-18T03:29:07.900Z	INFO	Starting workers	{"controller": "servicedescriptor", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ServiceDescriptor", "worker count": 1}
2023-10-18T03:29:07.901Z	INFO	DAG: |->{obj:*v1.Service, name: smarte-bxggqi-mysql, action: UPDATE}->{obj:*v1.Service, name: smarte-bxggqi-mysql-headless, action: UPDATE}->{obj:*v1.ConfigMap, name: smarte-bxggqi-mysql-rsm-env, action: UPDATE}->{obj:*v1.StatefulSet, name: smarte-bxggqi-mysql, action: UPDATE}->{obj:*v1alpha1.ReplicatedStateMachine, name: smarte-bxggqi-mysql, action: STATUS}	{"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"smarte-bxggqi-mysql","namespace":"ns-zpnclfdcyx"}, "namespace": "ns-zpnclfdcyx", "name": "smarte-bxggqi-mysql", "reconcileID": "0b2a9b64-37f9-4a76-92b7-07cf31d015ee", "ReplicatedStateMachine": {"name":"smarte-bxggqi-mysql","namespace":"ns-zpnclfdcyx"}}
2023-10-18T03:29:07.910Z	INFO	Starting workers	{"controller": "storageprovider", "controllerGroup": "storage.kubeblocks.io", "controllerKind": "StorageProvider", "worker count": 1}
2023-10-18T03:29:07.917Z	INFO	Starting workers	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "worker count": 1}
2023-10-18T03:29:07.942Z	INFO	parser Readiness probe event message failed	{"controller": "event", "controllerGroup": "", "controllerKind": "Event", "Event": {"name":"etcd-brqmiq-etcd-0.178f151ee4cfce82","namespace":"ns-wjxtiplupl"}, "namespace": "ns-wjxtiplupl", "name": "etcd-brqmiq-etcd-0.178f151ee4cfce82", "reconcileID": "f19b226d-af75-403c-9745-ec77b8dc106f", "event": {"name":"etcd-brqmiq-etcd-0.178f151ee4cfce82","namespace":"ns-wjxtiplupl"}, "message": "Pulling image \"apecloud/kubeblocks-tools:latest\""}
2023-10-18T03:29:07.942Z	INFO	parse probe event message failed	{"controller": "event", "controllerGroup": "", "controllerKind": "Event", "Event": {"name":"etcd-brqmiq-etcd-0.178f151ee4cfce82","namespace":"ns-wjxtiplupl"}, "namespace": "ns-wjxtiplupl", "name": "etcd-brqmiq-etcd-0.178f151ee4cfce82", "reconcileID": "f19b226d-af75-403c-9745-ec77b8dc106f", "event": {"name":"etcd-brqmiq-etcd-0.178f151ee4cfce82","namespace":"ns-wjxtiplupl"}, "message": "Pulling image \"apecloud/kubeblocks-tools:latest\""}
2023-10-18T03:29:07.988Z	INFO	class general-64c256g does not conform to any constraints	{"controller": "componentclassdefinition", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ComponentClassDefinition", "ComponentClassDefinition": {"name":"kb.classes.default.apecloud-mysql.mysql"}, "namespace": "", "name": "kb.classes.default.apecloud-mysql.mysql", "reconcileID": "24f18557-bce1-4465-b1fa-7c1b1bff9380", "classDefinition": {"name":"kb.classes.default.apecloud-mysql.mysql"}}
2023-10-18T03:29:07.990Z	INFO	class mo-48c384g does not conform to any constraints	{"controller": "componentclassdefinition", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ComponentClassDefinition", "ComponentClassDefinition": {"name":"kb.classes.default.mysql.mysql"}, "namespace": "", "name": "kb.classes.default.mysql.mysql", "reconcileID": "711ba827-3ca6-4bb0-8957-20ec4e9ea76a", "classDefinition": {"name":"kb.classes.default.mysql.mysql"}}
2023-10-18T03:29:07.990Z	INFO	class custom-0.1c0.5g does not conform to any constraints	{"controller": "componentclassdefinition", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ComponentClassDefinition", "ComponentClassDefinition": {"name":"kb.classes.custom.apecloud-mysql.mysql"}, "namespace": "", "name": "kb.classes.custom.apecloud-mysql.mysql", "reconcileID": "078105af-f08e-4855-9725-54aea380f04e", "classDefinition": {"name":"kb.classes.custom.apecloud-mysql.mysql"}}
2023-10-18T03:29:08.037Z	INFO	KubeAPIWarningLogger	unknown field "spec.service.metadata.creationTimestamp"
2023-10-18T03:29:08.037Z	INFO	KubeAPIWarningLogger	unknown field "spec.template.metadata.creationTimestamp"
2023-10-18T03:29:08.037Z	INFO	KubeAPIWarningLogger	unknown field "spec.volumeClaimTemplates[0].metadata.creationTimestamp"
2023-10-18T03:29:08.050Z	INFO	action list: []
	{"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"etcd-brqmiq-etcd","namespace":"ns-wjxtiplupl"}, "namespace": "ns-wjxtiplupl", "name": "etcd-brqmiq-etcd", "reconcileID": "ef435722-122c-4387-a9ba-9c889fad205c", "ReplicatedStateMachine": {"name":"etcd-brqmiq-etcd","namespace":"ns-wjxtiplupl"}}
2023-10-18T03:29:08.050Z	INFO	DAG: |->{obj:*v1.ConfigMap, name: etcd-brqmiq-etcd-rsm-env, action: UPDATE}->{obj:*v1.Service, name: etcd-brqmiq-etcd, action: UPDATE}->{obj:*v1.Service, name: etcd-brqmiq-etcd-headless, action: UPDATE}->{obj:*v1.StatefulSet, name: etcd-brqmiq-etcd, action: UPDATE}->{obj:*v1alpha1.ReplicatedStateMachine, name: etcd-brqmiq-etcd, action: STATUS}	{"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"etcd-brqmiq-etcd","namespace":"ns-wjxtiplupl"}, "namespace": "ns-wjxtiplupl", "name": "etcd-brqmiq-etcd", "reconcileID": "ef435722-122c-4387-a9ba-9c889fad205c", "ReplicatedStateMachine": {"name":"etcd-brqmiq-etcd","namespace":"ns-wjxtiplupl"}}
2023-10-18T03:29:08.146Z	INFO	DAG: |->{obj:*v1.Service, name: starrock-namtfe-fe-headless, action: UPDATE}->{obj:*v1.Service, name: starrock-namtfe-fe, action: UPDATE}->{obj:*v1.ConfigMap, name: starrock-namtfe-fe-rsm-env, action: UPDATE}->{obj:*v1.StatefulSet, name: starrock-namtfe-fe, action: UPDATE}->{obj:*v1alpha1.ReplicatedStateMachine, name: starrock-namtfe-fe, action: STATUS}	{"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"starrock-namtfe-fe","namespace":"ns-kdvgxqtdjd"}, "namespace": "ns-kdvgxqtdjd", "name": "starrock-namtfe-fe", "reconcileID": "78fbc70f-7056-4d37-9a46-d1cdef26dd8c", "ReplicatedStateMachine": {"name":"starrock-namtfe-fe","namespace":"ns-kdvgxqtdjd"}}
2023-10-18T03:29:08.287Z	INFO	DAG: |->{obj:*v1alpha1.ReplicatedStateMachine, name: ocbase-uyzsvj-ob-bundle, action: DELETE}	{"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"ocbase-uyzsvj-ob-bundle","namespace":"ns-zbwsxhdejh"}, "namespace": "ns-zbwsxhdejh", "name": "ocbase-uyzsvj-ob-bundle", "reconcileID": "8935aaaf-de32-4251-a14c-8fd10ffd17b5", "ReplicatedStateMachine": {"name":"ocbase-uyzsvj-ob-bundle","namespace":"ns-zbwsxhdejh"}}
2023-10-18T03:29:08.297Z	INFO	Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference	{"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"starrock-namtfe-be","namespace":"ns-kdvgxqtdjd"}, "namespace": "ns-kdvgxqtdjd", "name": "starrock-namtfe-be", "reconcileID": "40e6496f-c4c4-45bd-81c1-712cff201f62"}
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0xb1e329]

goroutine 722 [running]:
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:115 +0x1e5
panic({0x1f24760?, 0x3909dc0?})
	/usr/local/go/src/runtime/panic.go:914 +0x21f
k8s.io/api/core/v1.(*Service).GetName(0x275b6c0?)
	<autogenerated>:1 +0x9
github.com/apecloud/kubeblocks/internal/controller/model.(*realGraphClient).findMatchedVertex(0x7f567e19bfe8?, 0x7f57c546da68?, {0x275b6c0, 0x0})
	/src/internal/controller/model/graph_client.go:247 +0x10a
github.com/apecloud/kubeblocks/internal/controller/model.(*realGraphClient).DependOn(0xc0047a74c0?, 0xc0047a71f8?, {0x275b490?, 0xc003da2000?}, {0xc002f98900, 0x3, 0x0?})
	/src/internal/controller/model/graph_client.go:182 +0x9d
github.com/apecloud/kubeblocks/internal/controller/rsm.(*ObjectGenerationTransformer).Transform.func4(...)
	/src/internal/controller/rsm/transformer_object_generation.go:128
github.com/apecloud/kubeblocks/internal/controller/rsm.(*ObjectGenerationTransformer).Transform(0xc0046e0300?, {0x273e290?, 0xc003894c00?}, 0xc003fa73d0)
	/src/internal/controller/rsm/transformer_object_generation.go:138 +0xd62
github.com/apecloud/kubeblocks/internal/controller/graph.TransformerChain.ApplyTo({0xc0038971f0?, 0x7, 0x7?}, {0x273e290, 0xc003894c00}, 0x6?)
	/src/internal/controller/graph/transformer.go:57 +0x9b
github.com/apecloud/kubeblocks/internal/controller/rsm.(*PlanBuilder).Build(0xc00278e6e0)
	/src/internal/controller/rsm/plan_builder.go:84 +0xa6
github.com/apecloud/kubeblocks/controllers/workloads.(*ReplicatedStateMachineReconciler).Reconcile(0xc000a71d70, {0x273dc70, 0xc0038c9ad0}, {{{0xc000f94230, 0xd}, {0xc000fe1b18, 0x12}}})
	/src/controllers/workloads/replicatedstatemachine_controller.go:117 +0x6ee
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x273dc70?, {0x273dc70?, 0xc0038c9ad0?}, {{{0xc000f94230?, 0x1de9360?}, {0xc000fe1b18?, 0x272d7c0?}}})
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:118 +0xb7
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc00047da40, {0x273dca8, 0xc0001b4d20}, {0x2008120?, 0xc001a58040?})
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:314 +0x365
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc00047da40, {0x273dca8, 0xc0001b4d20})
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:265 +0x1c9
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:226 +0x79
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 128
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222 +0x565

Expected behavior
kubeblocks is ok.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

  • OS: [e.g. iOS]
  • Browser [e.g. chrome, safari]
  • Version [e.g. 22]

Additional context
Add any other context about the problem here.
@JashBook JashBook added kind/bug Something isn't working severity/major Great chance user will encounter the same problem labels Oct 18, 2023
@JashBook JashBook added this to the Release 0.7.0 milestone Oct 18, 2023
@JashBook JashBook added severity/critical Blocking or critical issues and removed severity/major Great chance user will encounter the same problem labels Oct 18, 2023
@free6om free6om mentioned this issue Oct 18, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@free6om free6om

Labels
bug ci kind/bug Something isn't working severity/critical Blocking or critical issues
Projects
None yet
Milestone
Release 0.7.0
Development

Successfully merging a pull request may close this issue.

fix: controller crash on nil service apecloud/kubeblocks
3 participants
@free6om @JashBook @apecloud-bot