Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] mysql volume-snapshot backup restore failed: cluster role is none #5389

Closed
JashBook opened this issue Oct 11, 2023 · 0 comments · Fixed by #5397
Closed

[BUG] mysql volume-snapshot backup restore failed: cluster role is none #5389

JashBook opened this issue Oct 11, 2023 · 0 comments · Fixed by #5397
Assignees
@wangyelei
Labels
bug kind/bug Something isn't working severity/major Great chance user will encounter the same problem
Milestone
Release 0.7.0

Comments

@JashBook
Copy link
Collaborator

JashBook commented Oct 11, 2023 •
edited
Loading

Describe the bug
mysql volume-snapshot backup restore failed: cluster role is none.

To Reproduce
Steps to reproduce the behavior:

  1. create mysql cluster
kbcli cluster create mysql mysql-test --mode raftGroup --availability-policy none
  1. do volume-snapshot backup
kbcli cluster backup mysql-test --method volume-snapshot
  1. restore backup
kbcli cluster restore mysql-backup --backup backup-default-mysql-test-20231011135941
  1. See error
kubectl get pod
NAME                                                              READY   STATUS              RESTARTS        AGE
mysql-backup-mysql-0                                              8/8     Running             2 (25s ago)     6m49s
mysql-backup-mysql-1                                              7/8     Error               1 (3m18s ago)   6m48s
mysql-backup-mysql-2                                              8/8     Running             2 (20s ago)     6m48s

kubectl get cluster 
NAME           CLUSTER-DEFINITION   VERSION           TERMINATION-POLICY   STATUS     AGE
mysql-backup   apecloud-mysql       ac-mysql-8.0.30   Delete               Updating   6m29s

describe error pod

kubectl describe pod mysql-backup-mysql-0
Name:             mysql-backup-mysql-0
Namespace:        default
Priority:         0
Service Account:  kb-mysql-backup
Node:             ip-172-31-1-64.us-west-2.compute.internal/172.31.1.64
Start Time:       Wed, 11 Oct 2023 14:01:33 +0800
Labels:           app.kubernetes.io/component=mysql
                  app.kubernetes.io/instance=mysql-backup
                  app.kubernetes.io/managed-by=kubeblocks
                  app.kubernetes.io/name=apecloud-mysql
                  app.kubernetes.io/version=ac-mysql-8.0.30
                  apps.kubeblocks.io/component-name=mysql
                  apps.kubeblocks.io/workload-type=Consensus
                  controller-revision-hash=mysql-backup-mysql-5767d9795b
                  statefulset.kubernetes.io/pod-name=mysql-backup-mysql-0
Annotations:      apps.kubeblocks.io/component-replicas: 3
Status:           Running
IP:               172.31.2.79
IPs:
  IP:           172.31.2.79
Controlled By:  StatefulSet/mysql-backup-mysql
Init Containers:
  role-agent-installer:
    Container ID:  containerd://b3bb30a44cc5f36c8d95b4a62ad2561002b752df0671b6017fb64fbcfa12ed4f
    Image:         msoap/shell2http:1.16.0
    Image ID:      docker.io/msoap/shell2http@sha256:a20bdde2f679de2cba6bf3d9f470489c7836d4d0d28232a2b295450809cd43ef
    Port:          <none>
    Host Port:     <none>
    Command:
      cp
      /app/shell2http
      /role-probe/agent
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Wed, 11 Oct 2023 14:01:37 +0800
      Finished:     Wed, 11 Oct 2023 14:01:37 +0800
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /role-probe from role-agent (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-vsxsg (ro)
Containers:
  mysql:
    Container ID:  containerd://507b3ad03209ce3671c642a28bbdd30365b30821d5781daffaaaf33983837b3b
    Image:         docker.io/apecloud/apecloud-mysql-server:8.0.30-5.beta2.20230830.g4ca1eb8.13
    Image ID:      docker.io/apecloud/apecloud-mysql-server@sha256:0f58f0cfac74cc502c7865c03ee53aead3bbb26fb9afb40bbef629fe47c0a3d6
    Ports:         3306/TCP, 13306/TCP
    Host Ports:    0/TCP, 0/TCP
    Command:
      /scripts/setup.sh
    State:          Running
      Started:      Wed, 11 Oct 2023 14:08:07 +0800
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Wed, 11 Oct 2023 14:04:47 +0800
      Finished:     Wed, 11 Oct 2023 14:07:56 +0800
    Ready:          True
    Restart Count:  2
    Limits:
      cpu:     500m
      memory:  512Mi
    Requests:
      cpu:     500m
      memory:  512Mi
    Environment Variables from:
      mysql-backup-mysql-env      ConfigMap  Optional: false
      mysql-backup-mysql-rsm-env  ConfigMap  Optional: false
    Environment:
      KB_POD_NAME:               mysql-backup-mysql-0 (v1:metadata.name)
      KB_POD_UID:                 (v1:metadata.uid)
      KB_NAMESPACE:              default (v1:metadata.namespace)
      KB_SA_NAME:                 (v1:spec.serviceAccountName)
      KB_NODENAME:                (v1:spec.nodeName)
      KB_HOST_IP:                 (v1:status.hostIP)
      KB_POD_IP:                  (v1:status.podIP)
      KB_POD_IPS:                 (v1:status.podIPs)
      KB_HOSTIP:                  (v1:status.hostIP)
      KB_PODIP:                   (v1:status.podIP)
      KB_PODIPS:                  (v1:status.podIPs)
      KB_CLUSTER_NAME:           mysql-backup
      KB_COMP_NAME:              mysql
      KB_CLUSTER_COMP_NAME:      mysql-backup-mysql
      KB_CLUSTER_UID_POSTFIX_8:  d991f221
      KB_POD_FQDN:               $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
      MYSQL_ROOT_HOST:           %
      MYSQL_ROOT_USER:           <set to the key 'username' in secret 'mysql-backup-conn-credential'>  Optional: false
      MYSQL_ROOT_PASSWORD:       <set to the key 'password' in secret 'mysql-backup-conn-credential'>  Optional: false
      MYSQL_DATABASE:            mydb
      MYSQL_USER:                u1
      MYSQL_PASSWORD:            u1
      CLUSTER_ID:                1
      CLUSTER_START_INDEX:       1
      REPLICATION_USER:          replicator
      REPLICATION_PASSWORD:      
      MYSQL_TEMPLATE_CONFIG:     
      MYSQL_CUSTOM_CONFIG:       
      MYSQL_DYNAMIC_CONFIG:      
      KB_EMBEDDED_WESQL:         1
    Mounts:
      /data/mysql from data (rw)
      /etc/annotations from annotations (rw)
      /opt/mysql from mysql-config (rw)
      /scripts from scripts (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-vsxsg (ro)
  metrics:
    Container ID:  containerd://590ed4c1b0e42efcf0059aeedfae5e7e5558598d637cb7afaba2be35cb795675
    Image:         docker.io/apecloud/agamotto:0.1.2-beta.1
    Image ID:      docker.io/apecloud/agamotto@sha256:cbab349b90490807a8d5039bf01bc7e37334f20c98c7dd75bc7fc4cf9e5b10ee
    Port:          9104/TCP
    Host Port:     0/TCP
    Command:
      /scripts/agamotto.sh
    State:          Running
      Started:      Wed, 11 Oct 2023 14:01:37 +0800
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     0
      memory:  0
    Requests:
      cpu:     0
      memory:  0
    Environment Variables from:
      mysql-backup-mysql-env      ConfigMap  Optional: false
      mysql-backup-mysql-rsm-env  ConfigMap  Optional: false
    Environment:
      KB_POD_NAME:               mysql-backup-mysql-0 (v1:metadata.name)
      KB_POD_UID:                 (v1:metadata.uid)
      KB_NAMESPACE:              default (v1:metadata.namespace)
      KB_SA_NAME:                 (v1:spec.serviceAccountName)
      KB_NODENAME:                (v1:spec.nodeName)
      KB_HOST_IP:                 (v1:status.hostIP)
      KB_POD_IP:                  (v1:status.podIP)
      KB_POD_IPS:                 (v1:status.podIPs)
      KB_HOSTIP:                  (v1:status.hostIP)
      KB_PODIP:                   (v1:status.podIP)
      KB_PODIPS:                  (v1:status.podIPs)
      KB_CLUSTER_NAME:           mysql-backup
      KB_COMP_NAME:              mysql
      KB_CLUSTER_COMP_NAME:      mysql-backup-mysql
      KB_CLUSTER_UID_POSTFIX_8:  d991f221
      KB_POD_FQDN:               $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
      DB_TYPE:                   MySQL
      ENDPOINT:                  localhost:3306
      MYSQL_USER:                <set to the key 'username' in secret 'mysql-backup-conn-credential'>  Optional: false
      MYSQL_PASSWORD:            <set to the key 'password' in secret 'mysql-backup-conn-credential'>  Optional: false
    Mounts:
      /data/mysql from data (rw)
      /opt/agamotto from agamotto-configuration (rw)
      /scripts from scripts (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-vsxsg (ro)
  vttablet:
    Container ID:  containerd://218b8f67a67502dab76bf79fcc6f54e17bcf052191b7cfcde03573d0385b3b79
    Image:         docker.io/apecloud/apecloud-mysql-scale:0.1.1
    Image ID:      docker.io/apecloud/apecloud-mysql-scale@sha256:dd4e22aefc096e29c057d464ed9552f550b3aad687dcf7469cb454ed94a6f319
    Ports:         15100/TCP, 16100/TCP
    Host Ports:    0/TCP, 0/TCP
    Command:
      /scripts/vttablet.sh
    State:          Running
      Started:      Wed, 11 Oct 2023 14:01:37 +0800
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     0
      memory:  0
    Requests:
      cpu:     0
      memory:  0
    Environment Variables from:
      mysql-backup-mysql-env      ConfigMap  Optional: false
      mysql-backup-mysql-rsm-env  ConfigMap  Optional: false
    Environment:
      KB_POD_NAME:               mysql-backup-mysql-0 (v1:metadata.name)
      KB_POD_UID:                 (v1:metadata.uid)
      KB_NAMESPACE:              default (v1:metadata.namespace)
      KB_SA_NAME:                 (v1:spec.serviceAccountName)
      KB_NODENAME:                (v1:spec.nodeName)
      KB_HOST_IP:                 (v1:status.hostIP)
      KB_POD_IP:                  (v1:status.podIP)
      KB_POD_IPS:                 (v1:status.podIPs)
      KB_HOSTIP:                  (v1:status.hostIP)
      KB_PODIP:                   (v1:status.podIP)
      KB_PODIPS:                  (v1:status.podIPs)
      KB_CLUSTER_NAME:           mysql-backup
      KB_COMP_NAME:              mysql
      KB_CLUSTER_COMP_NAME:      mysql-backup-mysql
      KB_CLUSTER_UID_POSTFIX_8:  d991f221
      KB_POD_FQDN:               $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
      CELL:                      zone1
      ETCD_SERVER:               $(KB_CLUSTER_NAME)-vtcontroller-headless
      ETCD_PORT:                 2379
      TOPOLOGY_FLAGS:            --topo_implementation etcd2 --topo_global_server_address $(ETCD_SERVER):$(ETCD_PORT) --topo_global_root /vitess/global
      VTTABLET_PORT:             15100
      VTTABLET_GRPC_PORT:        16100
      VTCTLD_HOST:               $(KB_CLUSTER_NAME)-vtcontroller-headless
      VTCTLD_WEB_PORT:           15000
      MYSQL_ROOT_USER:           <set to the key 'username' in secret 'mysql-backup-conn-credential'>  Optional: false
      MYSQL_ROOT_PASSWORD:       <set to the key 'password' in secret 'mysql-backup-conn-credential'>  Optional: false
    Mounts:
      /conf from mysql-scale-config (rw)
      /scripts from scripts (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-vsxsg (ro)
      /vtdataroot from data (rw)
  kb-we-syncer:
    Container ID:  containerd://91ff1559d16852308d78c29459bef5cfe7e19fc584316e2153f3b8caa5c6de0a
    Image:         docker.io/apecloud/kubeblocks-tools:0.7.0-beta.1
    Image ID:      docker.io/apecloud/kubeblocks-tools@sha256:f93b4193a87b7f3712ac771e4ed8a7c92907af2015717048927331c85530b7da
    Ports:         3501/TCP, 50001/TCP
    Host Ports:    0/TCP, 0/TCP
    Command:
      lorry
      --port
      3501
    State:          Running
      Started:      Wed, 11 Oct 2023 14:01:38 +0800
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     0
      memory:  0
    Requests:
      cpu:     0
      memory:  0
    Startup:   tcp-socket :3501 delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment Variables from:
      mysql-backup-mysql-env      ConfigMap  Optional: false
      mysql-backup-mysql-rsm-env  ConfigMap  Optional: false
    Environment:
      KB_POD_NAME:                mysql-backup-mysql-0 (v1:metadata.name)
      KB_POD_UID:                  (v1:metadata.uid)
      KB_NAMESPACE:               default (v1:metadata.namespace)
      KB_SA_NAME:                  (v1:spec.serviceAccountName)
      KB_NODENAME:                 (v1:spec.nodeName)
      KB_HOST_IP:                  (v1:status.hostIP)
      KB_POD_IP:                   (v1:status.podIP)
      KB_POD_IPS:                  (v1:status.podIPs)
      KB_HOSTIP:                   (v1:status.hostIP)
      KB_PODIP:                    (v1:status.podIP)
      KB_PODIPS:                   (v1:status.podIPs)
      KB_CLUSTER_NAME:            mysql-backup
      KB_COMP_NAME:               mysql
      KB_CLUSTER_COMP_NAME:       mysql-backup-mysql
      KB_CLUSTER_UID_POSTFIX_8:   d991f221
      KB_POD_FQDN:                $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
      KB_SERVICE_USER:            <set to the key 'username' in secret 'mysql-backup-conn-credential'>  Optional: false
      KB_SERVICE_PASSWORD:        <set to the key 'password' in secret 'mysql-backup-conn-credential'>  Optional: false
      KB_SERVICE_PORT:            3306
      KB_DATA_PATH:               /data/mysql
      KB_SERVICE_CHARACTER_TYPE:  mysql
      KB_WORKLOAD_TYPE:           Consensus
    Mounts:
      /data/mysql from data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-vsxsg (ro)
  config-manager:
    Container ID:  containerd://b363f20889533e464cb27098b0419266dc9a0394d45215b7fa1951ae5d66bc5e
    Image:         docker.io/apecloud/kubeblocks-tools:0.7.0-beta.1
    Image ID:      docker.io/apecloud/kubeblocks-tools@sha256:f93b4193a87b7f3712ac771e4ed8a7c92907af2015717048927331c85530b7da
    Port:          <none>
    Host Port:     <none>
    Command:
      env
    Args:
      PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:$(TOOLS_PATH)
      /bin/reloader
      --log-level
      info
      --operator-update-enable
      --tcp
      9901
      --config
      /opt/config-manager/config-manager.yaml
    State:          Running
      Started:      Wed, 11 Oct 2023 14:01:38 +0800
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     0
      memory:  0
    Requests:
      cpu:     0
      memory:  0
    Environment Variables from:
      mysql-backup-mysql-env      ConfigMap  Optional: false
      mysql-backup-mysql-rsm-env  ConfigMap  Optional: false
    Environment:
      KB_POD_NAME:               mysql-backup-mysql-0 (v1:metadata.name)
      KB_POD_UID:                 (v1:metadata.uid)
      KB_NAMESPACE:              default (v1:metadata.namespace)
      KB_SA_NAME:                 (v1:spec.serviceAccountName)
      KB_NODENAME:                (v1:spec.nodeName)
      KB_HOST_IP:                 (v1:status.hostIP)
      KB_POD_IP:                  (v1:status.podIP)
      KB_POD_IPS:                 (v1:status.podIPs)
      KB_HOSTIP:                  (v1:status.hostIP)
      KB_PODIP:                   (v1:status.podIP)
      KB_PODIPS:                  (v1:status.podIPs)
      KB_CLUSTER_NAME:           mysql-backup
      KB_COMP_NAME:              mysql
      KB_CLUSTER_COMP_NAME:      mysql-backup-mysql
      KB_CLUSTER_UID_POSTFIX_8:  d991f221
      KB_POD_FQDN:               $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
      CONFIG_MANAGER_POD_IP:      (v1:status.podIP)
      DB_TYPE:                   mysql
      MYSQL_USER:                <set to the key 'username' in secret 'mysql-backup-conn-credential'>  Optional: false
      MYSQL_PASSWORD:            <set to the key 'password' in secret 'mysql-backup-conn-credential'>  Optional: false
      DATA_SOURCE_NAME:          $(MYSQL_USER):$(MYSQL_PASSWORD)@(localhost:3306)/
      TOOLS_PATH:                /opt/kb-tools/reload/mysql-consensusset-config:/opt/config-manager
    Mounts:
      /conf from mysql-scale-config (rw)
      /opt/config-manager from config-manager-config (rw)
      /opt/kb-tools/reload/mysql-consensusset-config from cm-script-mysql-consensusset-config (rw)
      /opt/mysql from mysql-config (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-vsxsg (ro)
  action-0:
    Container ID:  containerd://121d276a5d2218dc539aa8ec0e2626382c46209a7091281cbe08417dae6903fb
    Image:         arey/mysql-client:latest
    Image ID:      docker.io/arey/mysql-client@sha256:3e80958e3c89a3d75971dc62a87876a5f35578a2adf93016b2d8cfb1fe10dfd9
    Port:          <none>
    Host Port:     <none>
    Command:
      /role-probe/agent
      -port
      36501
      -export-all-vars
      -form
      /role
      mysql -h127.0.0.1 -P3306 -u$KB_RSM_USERNAME -p$KB_RSM_PASSWORD -N -B -e "select role from information_schema.wesql_cluster_local" | xargs echo -n
    State:          Running
      Started:      Wed, 11 Oct 2023 14:01:38 +0800
    Ready:          True
    Restart Count:  0
    Environment:
      KB_RSM_USERNAME:  <set to the key 'username' in secret 'mysql-backup-conn-credential'>  Optional: false
      KB_RSM_PASSWORD:  <set to the key 'password' in secret 'mysql-backup-conn-credential'>  Optional: false
    Mounts:
      /role-probe from role-agent (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-vsxsg (ro)
  action-1:
    Container ID:  containerd://a4283ee4a329836c4e56f51f4de28d77bb9319803c151b4b0fd4a1ca9819d871
    Image:         busybox:latest
    Image ID:      docker.io/library/busybox@sha256:3fbc632167424a6d997e74f52b878d7cc478225cffac6bc977eedfe51c7f4e79
    Port:          <none>
    Host Port:     <none>
    Command:
      /role-probe/agent
      -port
      36502
      -export-all-vars
      -form
      /role
      echo $v_KB_RSM_LAST_STDOUT | tr '[:upper:]' '[:lower:]' | xargs echo -n
    State:          Running
      Started:      Wed, 11 Oct 2023 14:01:38 +0800
    Ready:          True
    Restart Count:  0
    Environment:
      KB_RSM_USERNAME:  <set to the key 'username' in secret 'mysql-backup-conn-credential'>  Optional: false
      KB_RSM_PASSWORD:  <set to the key 'password' in secret 'mysql-backup-conn-credential'>  Optional: false
    Mounts:
      /role-probe from role-agent (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-vsxsg (ro)
  kb-role-probe:
    Container ID:  containerd://6d7cc14aca20b136e1f535520324d99cbba43e88a2645fb53f9a7ab794d23dc6
    Image:         apecloud/kubeblocks-tools:latest
    Image ID:      docker.io/apecloud/kubeblocks-tools@sha256:376656b8a7973a138b3c879ba5cb4e588b2b9c06e0100096474ddb1b842ca2a4
    Port:          7373/TCP
    Host Port:     0/TCP
    Command:
      lorry
      --port
      7373
    State:          Running
      Started:      Wed, 11 Oct 2023 14:01:38 +0800
    Ready:          True
    Restart Count:  0
    Readiness:      http-get http://:7373/v1.0/bindings/custom%3Foperation=checkRole delay=0s timeout=1s period=1s #success=1 #failure=2
    Environment:
      KB_RSM_USERNAME:               <set to the key 'username' in secret 'mysql-backup-conn-credential'>  Optional: false
      KB_RSM_PASSWORD:               <set to the key 'password' in secret 'mysql-backup-conn-credential'>  Optional: false
      KB_RSM_ACTION_SVC_LIST:        [36501,36502]
      KB_SERVICE_USER:               <set to the key 'username' in secret 'mysql-backup-conn-credential'>  Optional: false
      KB_SERVICE_PASSWORD:           <set to the key 'password' in secret 'mysql-backup-conn-credential'>  Optional: false
      KB_RSM_SERVICE_PORT:           3306
      KB_SERVICE_PORT:               3306
      KB_RSM_ROLE_UPDATE_MECHANISM:  DirectAPIServerEventUpdate
      KB_RSM_ROLE_PROBE_TIMEOUT:     1
      KB_POD_NAME:                   mysql-backup-mysql-0 (v1:metadata.name)
      KB_NAMESPACE:                  default (v1:metadata.namespace)
      KB_POD_UID:                     (v1:metadata.uid)
      KB_NODENAME:                    (v1:spec.nodeName)
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-vsxsg (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  data-mysql-backup-mysql-0
    ReadOnly:   false
  annotations:
    Type:  DownwardAPI (a volume populated by information about the pod)
    Items:
      metadata.annotations['cs.apps.kubeblocks.io/leader'] -> leader
      metadata.annotations['apps.kubeblocks.io/component-replicas'] -> component-replicas
  agamotto-configuration:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      mysql-backup-mysql-agamotto-configuration
    Optional:  false
  scripts:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      mysql-backup-mysql-apecloud-mysql-scripts
    Optional:  false
  mysql-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      mysql-backup-mysql-mysql-consensusset-config
    Optional:  false
  mysql-scale-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      mysql-backup-mysql-vttablet-config
    Optional:  false
  cm-script-mysql-consensusset-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      sidecar-mysql-reload-script-mysql-backup
    Optional:  false
  config-manager-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      sidecar-mysql-backup-mysql-config-manager-config
    Optional:  false
  role-agent:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  kube-api-access-vsxsg:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 kb-data=true:NoSchedule
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age    From               Message
  ----     ------     ----   ----               -------
  Normal   Scheduled  7m38s  default-scheduler  Successfully assigned default/mysql-backup-mysql-0 to ip-172-31-1-64.us-west-2.compute.internal
  Normal   Pulled     7m34s  kubelet            Container image "msoap/shell2http:1.16.0" already present on machine
  Normal   Created    7m34s  kubelet            Created container role-agent-installer
  Normal   Started    7m34s  kubelet            Started container role-agent-installer
  Normal   Pulled     7m34s  kubelet            Container image "docker.io/apecloud/apecloud-mysql-server:8.0.30-5.beta2.20230830.g4ca1eb8.13" already present on machine
  Normal   Created    7m34s  kubelet            Created container mysql
  Normal   Started    7m34s  kubelet            Started container mysql
  Normal   Pulled     7m34s  kubelet            Container image "docker.io/apecloud/agamotto:0.1.2-beta.1" already present on machine
  Normal   Created    7m34s  kubelet            Created container metrics
  Normal   Started    7m34s  kubelet            Started container metrics
  Normal   Pulled     7m34s  kubelet            Container image "docker.io/apecloud/apecloud-mysql-scale:0.1.1" already present on machine
  Normal   Created    7m34s  kubelet            Created container vttablet
  Normal   Started    7m34s  kubelet            Started container vttablet
  Normal   Pulled     7m34s  kubelet            Container image "docker.io/apecloud/kubeblocks-tools:0.7.0-beta.1" already present on machine
  Normal   Created    7m34s  kubelet            Created container kb-we-syncer
  Normal   Started    7m33s  kubelet            Started container kb-we-syncer
  Normal   Pulled     7m33s  kubelet            Container image "docker.io/apecloud/kubeblocks-tools:0.7.0-beta.1" already present on machine
  Normal   Created    7m33s  kubelet            Created container config-manager
  Normal   Started    7m33s  kubelet            Started container config-manager
  Normal   Pulled     7m33s  kubelet            Container image "arey/mysql-client:latest" already present on machine
  Normal   Created    7m33s  kubelet            Created container action-0
  Normal   Started    7m33s  kubelet            Started container action-0
  Normal   Pulled     7m33s  kubelet            Container image "busybox:latest" already present on machine
  Normal   Created    7m33s  kubelet            Created container action-1
  Normal   Started    7m33s  kubelet            Started container action-1
  Normal   Pulled     7m33s  kubelet            Container image "apecloud/kubeblocks-tools:latest" already present on machine
  Warning  BackOff    75s    kubelet            Back-off restarting failed container

describe cluster

kubectl describe cluster mysql-backup
Name:         mysql-backup
Namespace:    default
Labels:       clusterdefinition.kubeblocks.io/name=apecloud-mysql
              clusterversion.kubeblocks.io/name=ac-mysql-8.0.30
Annotations:  kubeblocks.io/restore-from-backup:
                {"mysql":{"name":"backup-default-mysql-test-20231011135941","namespace":"default","managementPolicy":"Parallel"}}
API Version:  apps.kubeblocks.io/v1alpha1
Kind:         Cluster
Metadata:
  Creation Timestamp:  2023-10-11T06:01:17Z
  Finalizers:
    cluster.kubeblocks.io/finalizer
  Generation:        1
  Resource Version:  123552
  UID:               ba586834-8a29-4659-a625-c1ddd991f221
Spec:
  Affinity:
    Pod Anti Affinity:     Required
    Tenancy:               SharedNode
  Cluster Definition Ref:  apecloud-mysql
  Cluster Version Ref:     ac-mysql-8.0.30
  Component Specs:
    Component Def Ref:  mysql
    Enabled Logs:
      slow
      error
    Monitor:        false
    Name:           mysql
    No Create PDB:  false
    Replicas:       3
    Resources:
      Limits:
        Cpu:     500m
        Memory:  512Mi
      Requests:
        Cpu:     500m
        Memory:  512Mi
    Volume Claim Templates:
      Name:  data
      Spec:
        Access Modes:
          ReadWriteOnce
        Resources:
          Requests:
            Storage:  20Gi
  Monitor:
  Resources:
    Cpu:     0
    Memory:  0
  Storage:
    Size:              0
  Termination Policy:  Delete
Status:
  Cluster Def Generation:  2
  Components:
    Mysql:
      Consensus Set Status:
        Leader:
          Access Mode:  None
          Name:         
          Pod:          Unknown
      Message:
        Pod/mysql-backup-mysql-1:  Role probe timeout, check whether the application is available
        Pod/mysql-backup-mysql-2:  Role probe timeout, check whether the application is available
      Phase:                       Failed
      Pods Ready:                  false
  Conditions:
    Last Transition Time:  2023-10-11T06:01:17Z
    Message:               The operator has started the provisioning of Cluster: mysql-backup
    Observed Generation:   1
    Reason:                PreCheckSucceed
    Status:                True
    Type:                  ProvisioningStarted
    Last Transition Time:  2023-10-11T06:01:17Z
    Message:               Successfully applied for resources
    Observed Generation:   1
    Reason:                ApplyResourcesSucceed
    Status:                True
    Type:                  ApplyResources
    Last Transition Time:  2023-10-11T06:01:33Z
    Message:               pods are not ready in Components: [mysql], refer to related component message in Cluster.status.components
    Reason:                ReplicasNotReady
    Status:                False
    Type:                  ReplicasReady
    Last Transition Time:  2023-10-11T06:01:33Z
    Message:               pods are unavailable in Components: [mysql], refer to related component message in Cluster.status.components
    Reason:                ComponentsNotReady
    Status:                False
    Type:                  Ready
  Observed Generation:     1
  Phase:                   Failed
Events:
  Type     Reason                    Age                    From                Message
  ----     ------                    ----                   ----                -------
  Normal   NeedWaiting               6m49s (x3 over 6m49s)  cluster-controller  waiting for restore "mysql-backup-mysql-ba586834-preparedata" successfully
  Normal   PreCheckSucceed           6m49s                  cluster-controller  The operator has started the provisioning of Cluster: mysql-backup
  Normal   ApplyResourcesSucceed     6m49s                  cluster-controller  Successfully applied for resources
  Normal   ComponentPhaseTransition  6m33s                  cluster-controller  Create a new component
  Normal   ComponentPhaseTransition  3m3s                   cluster-controller  component is Updating
  Normal   ComponentPhaseTransition  8s (x2 over 3m19s)     cluster-controller  component is Failed
  Warning  Failed                    8s (x2 over 3m19s)     cluster-controller  Cluster: mysql-backup is Failed, check according to the components message

logs pod

kubectl logs mysql-backup-mysql-0 --previous 
Defaulted container "mysql" out of: mysql, metrics, vttablet, kb-we-syncer, config-manager, action-0, action-1, kb-role-probe, role-agent-installer (init)
2023-10-11 06:01:37+00:00 [Note] [MySQL]: Entrypoint script for MySQL Server  started.
2023-10-11 06:01:37+00:00 [Note] [MySQL]: KB_EMBEDDED_WESQL=1
2023-10-11 06:01:37+00:00 [Note] [KubeBlocks]: 
2023-10-11 06:01:37+00:00 [Note] [KubeBlocks]: Welcome to the KubeBlocks MySQL Cluster Container
2023-10-11 06:01:37+00:00 [Note] [KubeBlocks]: 
2023-10-11 06:01:37+00:00 [Note] [KubeBlocks]: KB_VOLUME_DIR=/data/mysql
2023-10-11 06:01:37+00:00 [Note] [KubeBlocks]: KB_DATA_DIR=/data/mysql/data
2023-10-11 06:01:37+00:00 [Note] [KubeBlocks]: KB_LOG_DIR=/data/mysql/log
2023-10-11 06:01:37+00:00 [Note] [KubeBlocks]: KB_CLUSTER_LEADER=
2023-10-11 06:01:37+00:00 [Note] [KubeBlocks]: KB_CLUSTER_FOLLOWERS=
2023-10-11 06:01:37+00:00 [Note] [KubeBlocks]: KB_CLUSTER_UID=6593cebb-7975-4a47-a51e-1c42bd3d7486
2023-10-11 06:01:37+00:00 [Note] [KubeBlocks]: KB_CLUSTER_UID_PATH=/data/mysql/.kb_cluster_uid
2023-10-11 06:01:37+00:00 [Note] [KubeBlocks]: KB_MYSQL_CONF_FILE=/opt/mysql/my.cnf
2023-10-11 06:01:37+00:00 [Note] [KubeBlocks]: KB_MYSQL_PORT=3306
2023-10-11 06:01:37+00:00 [Note] [KubeBlocks]: KB_POD_NAME=mysql-backup-mysql-0
2023-10-11 06:01:37+00:00 [Note] [KubeBlocks]: KB_POD_N=3
2023-10-11 06:01:37+00:00 [Note] [KubeBlocks]: KB_POD_INDEX=0
2023-10-11 06:01:37+00:00 [Note] [KubeBlocks]: KB_POD_HOST=mysql-backup-mysql-0.mysql-backup-mysql-headless
2023-10-11 06:01:37+00:00 [Note] [KubeBlocks]: 
2023-10-11 06:01:37+00:00 [Note] [KubeBlocks]: recreate from existing volumes, touch /data/mysql/data/.resetup_db
2023-10-11 06:01:37+00:00 [Note] [KubeBlocks]: 
2023-10-11 06:01:37+00:00 [Note] [KubeBlocks]: CLUSTER_INFO=mysql-backup-mysql-0.mysql-backup-mysql-headless:13306;mysql-backup-mysql-1.mysql-backup-mysql-headless:13306;mysql-backup-mysql-2.mysql-backup-mysql-headless:13306@1
2023-10-11 06:01:37+00:00 [Note] [KubeBlocks]: CLUSTER_LEADER_HOST=
2023-10-11 06:01:37+00:00 [Note] [KubeBlocks]: 
2023-10-11 06:01:37+00:00 [Note] [KubeBlocks]: MYSQL_EXTRA_CONF_FILE=/opt/mysql/my.cnf
2023-10-11 06:01:37+00:00 [Note] [KubeBlocks]: 
2023-10-11 06:01:37+00:00 [Note] [KubeBlocks]: Injecting configuration /opt/mysql/my.cnf to /etc/my.cnf
2023-10-11 06:01:37+00:00 [Note] [KubeBlocks]: Injecting configuration /etc/mysql/wesql_cluster.cnf to /etc/my.cnf
2023-10-11 06:01:37+00:00 [Note] [KubeBlocks]: DATADIR=/data/mysql/data/
2023-10-11 06:01:38+00:00 [Note] [KubeBlocks]: PID_FILE=/var/run/mysqld/mysqld.pid
2023-10-11 06:01:38+00:00 [Note] [KubeBlocks]: SOCKET=/var/run/mysqld/mysqld.sock
2023-10-11 06:01:38+00:00 [Note] [KubeBlocks]: Generating post logrotate shell script: /apecloud/mysql/scripts//post_logroate.sh
2023-10-11 06:01:38+00:00 [Note] [KubeBlocks]: error_log=/data/mysql/log/mysqld-error.log
2023-10-11 06:01:38+00:00 [Note] [KubeBlocks]: slow_log=/data/mysql/log/mysqld-slowquery.log
2023-10-11 06:01:38+00:00 [Note] [KubeBlocks]: general_log=/data/mysql/data/mysql-backup-mysql-0.log
2023-10-11 06:01:38+00:00 [Note] [KubeBlocks]: Switching to dedicated user 'mysql'
2023-10-11 06:01:38+00:00 [Note] [MySQL]: Entrypoint script for MySQL Server  started.
2023-10-11 06:01:38+00:00 [Note] [MySQL]: KB_EMBEDDED_WESQL=1
2023-10-11 06:01:39+00:00 [Note] [KubeBlocks]: DATADIR=/data/mysql/data/
2023-10-11 06:01:39+00:00 [Note] [KubeBlocks]: PID_FILE=/var/run/mysqld/mysqld.pid
2023-10-11 06:01:39+00:00 [Note] [KubeBlocks]: SOCKET=/var/run/mysqld/mysqld.sock
2023-10-11 06:01:39+00:00 [Note] [KubeBlocks]: JEMALLOC_ENABLED=1
2023-10-11 06:01:39+00:00 [Note] [KubeBlocks]: Using jemalloc: LD_PRELOAD=/usr/local/lib/libjemalloc.so.2
2023-10-11 06:01:39+00:00 [Note] [KubeBlocks]: Using persisted data
2023-10-11 06:01:39+00:00 [Note] [KubeBlocks]: Found restore file: /data/mysql/data//.restore_new_cluster
2023-10-11 06:01:39+00:00 [Note] [KubeBlocks]: Rebuilding new cluster from backup
2023-10-11 06:01:39+00:00 [Note] [KubeBlocks]: Initializing consensus meta: 'mysqld --cluster-recover-snapshot --cluster-recover-new-cluster --cluster-info=mysql-backup-mysql-0.mysql-backup-mysql-headless:13306;mysql-backup-mysql-1.mysql-backup-mysql-headless:13306;mysql-backup-mysql-2.mysql-backup-mysql-headless:13306@1 --cluster-id=1 --cluster-start-index=1 --cluster-force-change-meta=ON --upgrade-wesql'
2023-10-11 06:01:42+00:00 [Note] [KubeBlocks]: Start temporary server in background with backup/snapshot recovery
2023-10-11 06:01:42+00:00 [Note] [KubeBlocks]: Starting temporary server in background: 'mysqld --skip-grant-tables --cluster-recover-snapshot --cluster-recover-new-cluster'
2023-10-11 06:01:42+00:00 [Note] [KubeBlocks]: Waiting for server startup
2023-10-11 06:01:42+00:00 [Warn] [KubeBlocks]: The temporary server is still not started.
2023-10-11 06:01:43+00:00 [Warn] [KubeBlocks]: The temporary server is still not started.
2023-10-11 06:01:44+00:00 [Warn] [KubeBlocks]: The temporary server is still not started.
...
2023-10-11 06:04:43+00:00 [Warn] [KubeBlocks]: The temporary server is still not started.
2023-10-11 06:04:44+00:00 [Warn] [KubeBlocks]: The temporary server is still not started.
2023-10-11 06:04:45+00:00 [Warn] [KubeBlocks]: The temporary server is still not started.
2023-10-11 06:04:46+00:00 [ERROR] [KubeBlocks]: Unable to start server.
2023-10-11 06:04:46+00:00 [Note] [KubeBlocks]: Stopping temporary server(pid: 269).
mysqladmin: connect to server at 'localhost' failed
error: 'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)'
Check that mysqld is running and that the socket: '/var/run/mysqld/mysqld.sock' exists!
2023-10-11 06:04:46+00:00 [ERROR] [KubeBlocks]: Unable to shut down server.

kubectl logs mysql-backup-mysql-1 --previous
Defaulted container "mysql" out of: mysql, metrics, vttablet, kb-we-syncer, config-manager, action-0, action-1, kb-role-probe, role-agent-installer (init)
2023-10-11 06:01:56+00:00 [Note] [MySQL]: Entrypoint script for MySQL Server  started.
2023-10-11 06:01:56+00:00 [Note] [MySQL]: KB_EMBEDDED_WESQL=1
2023-10-11 06:01:56+00:00 [Note] [KubeBlocks]: 
2023-10-11 06:01:56+00:00 [Note] [KubeBlocks]: Welcome to the KubeBlocks MySQL Cluster Container
2023-10-11 06:01:56+00:00 [Note] [KubeBlocks]: 
2023-10-11 06:01:56+00:00 [Note] [KubeBlocks]: KB_VOLUME_DIR=/data/mysql
2023-10-11 06:01:56+00:00 [Note] [KubeBlocks]: KB_DATA_DIR=/data/mysql/data
2023-10-11 06:01:57+00:00 [Note] [KubeBlocks]: KB_LOG_DIR=/data/mysql/log
2023-10-11 06:01:57+00:00 [Note] [KubeBlocks]: KB_CLUSTER_LEADER=
2023-10-11 06:01:57+00:00 [Note] [KubeBlocks]: KB_CLUSTER_FOLLOWERS=
2023-10-11 06:01:57+00:00 [Note] [KubeBlocks]: KB_CLUSTER_UID=6593cebb-7975-4a47-a51e-1c42bd3d7486
2023-10-11 06:01:57+00:00 [Note] [KubeBlocks]: KB_CLUSTER_UID_PATH=/data/mysql/.kb_cluster_uid
2023-10-11 06:01:57+00:00 [Note] [KubeBlocks]: KB_MYSQL_CONF_FILE=/opt/mysql/my.cnf
2023-10-11 06:01:57+00:00 [Note] [KubeBlocks]: KB_MYSQL_PORT=3306
2023-10-11 06:01:57+00:00 [Note] [KubeBlocks]: KB_POD_NAME=mysql-backup-mysql-1
2023-10-11 06:01:57+00:00 [Note] [KubeBlocks]: KB_POD_N=3
2023-10-11 06:01:57+00:00 [Note] [KubeBlocks]: KB_POD_INDEX=1
2023-10-11 06:01:57+00:00 [Note] [KubeBlocks]: KB_POD_HOST=mysql-backup-mysql-1.mysql-backup-mysql-headless
2023-10-11 06:01:57+00:00 [Note] [KubeBlocks]: 
2023-10-11 06:01:57+00:00 [Note] [KubeBlocks]: recreate from existing volumes, touch /data/mysql/data/.resetup_db
2023-10-11 06:01:57+00:00 [Note] [KubeBlocks]: 
2023-10-11 06:01:57+00:00 [Note] [KubeBlocks]: CLUSTER_INFO=mysql-backup-mysql-0.mysql-backup-mysql-headless:13306;mysql-backup-mysql-1.mysql-backup-mysql-headless:13306;mysql-backup-mysql-2.mysql-backup-mysql-headless:13306@2
2023-10-11 06:01:57+00:00 [Note] [KubeBlocks]: CLUSTER_LEADER_HOST=
2023-10-11 06:01:57+00:00 [Note] [KubeBlocks]: 
2023-10-11 06:01:57+00:00 [Note] [KubeBlocks]: MYSQL_EXTRA_CONF_FILE=/opt/mysql/my.cnf
2023-10-11 06:01:57+00:00 [Note] [KubeBlocks]: 
2023-10-11 06:01:57+00:00 [Note] [KubeBlocks]: Injecting configuration /opt/mysql/my.cnf to /etc/my.cnf
2023-10-11 06:01:57+00:00 [Note] [KubeBlocks]: Injecting configuration /etc/mysql/wesql_cluster.cnf to /etc/my.cnf
2023-10-11 06:01:57+00:00 [Note] [KubeBlocks]: DATADIR=/data/mysql/data/
2023-10-11 06:01:57+00:00 [Note] [KubeBlocks]: PID_FILE=/var/run/mysqld/mysqld.pid
2023-10-11 06:01:57+00:00 [Note] [KubeBlocks]: SOCKET=/var/run/mysqld/mysqld.sock
2023-10-11 06:01:58+00:00 [Note] [KubeBlocks]: Generating post logrotate shell script: /apecloud/mysql/scripts//post_logroate.sh
2023-10-11 06:01:58+00:00 [Note] [KubeBlocks]: error_log=/data/mysql/log/mysqld-error.log
2023-10-11 06:01:58+00:00 [Note] [KubeBlocks]: slow_log=/data/mysql/log/mysqld-slowquery.log
2023-10-11 06:01:58+00:00 [Note] [KubeBlocks]: general_log=/data/mysql/data/mysql-backup-mysql-1.log
2023-10-11 06:01:58+00:00 [Note] [KubeBlocks]: Switching to dedicated user 'mysql'
2023-10-11 06:01:58+00:00 [Note] [MySQL]: Entrypoint script for MySQL Server  started.
2023-10-11 06:01:58+00:00 [Note] [MySQL]: KB_EMBEDDED_WESQL=1
2023-10-11 06:01:58+00:00 [Note] [KubeBlocks]: DATADIR=/data/mysql/data/
2023-10-11 06:01:58+00:00 [Note] [KubeBlocks]: PID_FILE=/var/run/mysqld/mysqld.pid
2023-10-11 06:01:58+00:00 [Note] [KubeBlocks]: SOCKET=/var/run/mysqld/mysqld.sock
2023-10-11 06:01:59+00:00 [Note] [KubeBlocks]: JEMALLOC_ENABLED=1
2023-10-11 06:01:59+00:00 [Note] [KubeBlocks]: Using jemalloc: LD_PRELOAD=/usr/local/lib/libjemalloc.so.2
2023-10-11 06:01:59+00:00 [Note] [KubeBlocks]: Using persisted data
2023-10-11 06:01:59+00:00 [Note] [KubeBlocks]: Starting temporary server in background: 'mysqld --skip-grant-tables --upgrade-wesql'
2023-10-11 06:01:59+00:00 [Note] [KubeBlocks]: Waiting for server startup
2023-10-11 06:01:59+00:00 [Warn] [KubeBlocks]: The temporary server is still not started.
2023-10-11 06:02:00+00:00 [Warn] [KubeBlocks]: The temporary server is still not started.
2023-10-11 06:02:01+00:00 [Warn] [KubeBlocks]: The temporary server is still not started.
...
2023-10-11 06:05:00+00:00 [Warn] [KubeBlocks]: The temporary server is still not started.
2023-10-11 06:05:01+00:00 [Warn] [KubeBlocks]: The temporary server is still not started.
2023-10-11 06:05:02+00:00 [Warn] [KubeBlocks]: The temporary server is still not started.
2023-10-11 06:05:03+00:00 [ERROR] [KubeBlocks]: Unable to start server.
2023-10-11 06:05:03+00:00 [Note] [KubeBlocks]: Stopping temporary server(pid: 217).
mysqladmin: connect to server at 'localhost' failed
error: 'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)'
Check that mysqld is running and that the socket: '/var/run/mysqld/mysqld.sock' exists!
2023-10-11 06:05:03+00:00 [ERROR] [KubeBlocks]: Unable to shut down server.

kubectl logs mysql-backup-mysql-2 --previous
Defaulted container "mysql" out of: mysql, metrics, vttablet, kb-we-syncer, config-manager, action-0, action-1, kb-role-probe, role-agent-installer (init)
2023-10-11 06:01:46+00:00 [Note] [MySQL]: Entrypoint script for MySQL Server  started.
2023-10-11 06:01:46+00:00 [Note] [MySQL]: KB_EMBEDDED_WESQL=1
2023-10-11 06:01:46+00:00 [Note] [KubeBlocks]: 
2023-10-11 06:01:46+00:00 [Note] [KubeBlocks]: Welcome to the KubeBlocks MySQL Cluster Container
2023-10-11 06:01:46+00:00 [Note] [KubeBlocks]: 
2023-10-11 06:01:46+00:00 [Note] [KubeBlocks]: KB_VOLUME_DIR=/data/mysql
2023-10-11 06:01:46+00:00 [Note] [KubeBlocks]: KB_DATA_DIR=/data/mysql/data
2023-10-11 06:01:46+00:00 [Note] [KubeBlocks]: KB_LOG_DIR=/data/mysql/log
2023-10-11 06:01:46+00:00 [Note] [KubeBlocks]: KB_CLUSTER_LEADER=
2023-10-11 06:01:46+00:00 [Note] [KubeBlocks]: KB_CLUSTER_FOLLOWERS=
2023-10-11 06:01:46+00:00 [Note] [KubeBlocks]: KB_CLUSTER_UID=6593cebb-7975-4a47-a51e-1c42bd3d7486
2023-10-11 06:01:46+00:00 [Note] [KubeBlocks]: KB_CLUSTER_UID_PATH=/data/mysql/.kb_cluster_uid
2023-10-11 06:01:46+00:00 [Note] [KubeBlocks]: KB_MYSQL_CONF_FILE=/opt/mysql/my.cnf
2023-10-11 06:01:46+00:00 [Note] [KubeBlocks]: KB_MYSQL_PORT=3306
2023-10-11 06:01:46+00:00 [Note] [KubeBlocks]: KB_POD_NAME=mysql-backup-mysql-2
2023-10-11 06:01:46+00:00 [Note] [KubeBlocks]: KB_POD_N=3
2023-10-11 06:01:46+00:00 [Note] [KubeBlocks]: KB_POD_INDEX=2
2023-10-11 06:01:46+00:00 [Note] [KubeBlocks]: KB_POD_HOST=mysql-backup-mysql-2.mysql-backup-mysql-headless
2023-10-11 06:01:46+00:00 [Note] [KubeBlocks]: 
2023-10-11 06:01:46+00:00 [Note] [KubeBlocks]: recreate from existing volumes, touch /data/mysql/data/.resetup_db
2023-10-11 06:01:46+00:00 [Note] [KubeBlocks]: 
2023-10-11 06:01:46+00:00 [Note] [KubeBlocks]: CLUSTER_INFO=mysql-backup-mysql-0.mysql-backup-mysql-headless:13306;mysql-backup-mysql-1.mysql-backup-mysql-headless:13306;mysql-backup-mysql-2.mysql-backup-mysql-headless:13306@3
2023-10-11 06:01:46+00:00 [Note] [KubeBlocks]: CLUSTER_LEADER_HOST=
2023-10-11 06:01:46+00:00 [Note] [KubeBlocks]: 
2023-10-11 06:01:46+00:00 [Note] [KubeBlocks]: MYSQL_EXTRA_CONF_FILE=/opt/mysql/my.cnf
2023-10-11 06:01:46+00:00 [Note] [KubeBlocks]: 
2023-10-11 06:01:46+00:00 [Note] [KubeBlocks]: Injecting configuration /opt/mysql/my.cnf to /etc/my.cnf
2023-10-11 06:01:46+00:00 [Note] [KubeBlocks]: Injecting configuration /etc/mysql/wesql_cluster.cnf to /etc/my.cnf
2023-10-11 06:01:47+00:00 [Note] [KubeBlocks]: DATADIR=/data/mysql/data/
2023-10-11 06:01:47+00:00 [Note] [KubeBlocks]: PID_FILE=/var/run/mysqld/mysqld.pid
2023-10-11 06:01:47+00:00 [Note] [KubeBlocks]: SOCKET=/var/run/mysqld/mysqld.sock
2023-10-11 06:01:47+00:00 [Note] [KubeBlocks]: Generating post logrotate shell script: /apecloud/mysql/scripts//post_logroate.sh
2023-10-11 06:01:47+00:00 [Note] [KubeBlocks]: error_log=/data/mysql/log/mysqld-error.log
2023-10-11 06:01:47+00:00 [Note] [KubeBlocks]: slow_log=/data/mysql/log/mysqld-slowquery.log
2023-10-11 06:01:47+00:00 [Note] [KubeBlocks]: general_log=/data/mysql/data/mysql-backup-mysql-2.log
2023-10-11 06:01:47+00:00 [Note] [KubeBlocks]: Switching to dedicated user 'mysql'
2023-10-11 06:01:48+00:00 [Note] [MySQL]: Entrypoint script for MySQL Server  started.
2023-10-11 06:01:48+00:00 [Note] [MySQL]: KB_EMBEDDED_WESQL=1
2023-10-11 06:01:48+00:00 [Note] [KubeBlocks]: DATADIR=/data/mysql/data/
2023-10-11 06:01:48+00:00 [Note] [KubeBlocks]: PID_FILE=/var/run/mysqld/mysqld.pid
2023-10-11 06:01:48+00:00 [Note] [KubeBlocks]: SOCKET=/var/run/mysqld/mysqld.sock
2023-10-11 06:01:48+00:00 [Note] [KubeBlocks]: JEMALLOC_ENABLED=1
2023-10-11 06:01:48+00:00 [Note] [KubeBlocks]: Using jemalloc: LD_PRELOAD=/usr/local/lib/libjemalloc.so.2
2023-10-11 06:01:48+00:00 [Note] [KubeBlocks]: Using persisted data
2023-10-11 06:01:48+00:00 [Note] [KubeBlocks]: Starting temporary server in background: 'mysqld --skip-grant-tables --upgrade-wesql'
2023-10-11 06:01:48+00:00 [Note] [KubeBlocks]: Waiting for server startup
2023-10-11 06:01:48+00:00 [Warn] [KubeBlocks]: The temporary server is still not started.
2023-10-11 06:01:49+00:00 [Warn] [KubeBlocks]: The temporary server is still not started.
2023-10-11 06:01:51+00:00 [Warn] [KubeBlocks]: The temporary server is still not started.
2023-10-11 06:01:52+00:00 [Warn] [KubeBlocks]: The temporary server is still not started.
...
2023-10-11 06:04:49+00:00 [Warn] [KubeBlocks]: The temporary server is still not started.
2023-10-11 06:04:50+00:00 [Warn] [KubeBlocks]: The temporary server is still not started.
2023-10-11 06:04:51+00:00 [Warn] [KubeBlocks]: The temporary server is still not started.
2023-10-11 06:04:52+00:00 [Warn] [KubeBlocks]: The temporary server is still not started.
2023-10-11 06:04:53+00:00 [ERROR] [KubeBlocks]: Unable to start server.
2023-10-11 06:04:53+00:00 [Note] [KubeBlocks]: Stopping temporary server(pid: 217).
mysqladmin: connect to server at 'localhost' failed
error: 'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)'
Check that mysqld is running and that the socket: '/var/run/mysqld/mysqld.sock' exists!
2023-10-11 06:04:53+00:00 [ERROR] [KubeBlocks]: Unable to shut down server.

kubectl logs mysql-backup-mysql-0  kb-role-probe 
{"level":"error","ts":1697004098.7066927,"caller":"binding/operation_volume_protection.go:182","msg":"unmarshal volume protection spec error","raw spec":"","error":"unexpected end of JSON input","stacktrace":"github.com/apecloud/kubeblocks/lorry/binding.(*operationVolumeProtection).initVolumes\n\t/src/lorry/binding/operation_volume_protection.go:182\ngithub.com/apecloud/kubeblocks/lorry/binding.init.1\n\t/src/lorry/binding/operation_volume_protection.go:103\nruntime.doInit1\n\t/usr/local/go/src/runtime/proc.go:6740\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6707\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:249"}
{"level":"info","ts":1697004098.7070057,"caller":"binding/operation_volume_protection.go:104","msg":"init volumes to monitor failed","error":"unexpected end of JSON input"}
{"level":"info","ts":1697004098.7071304,"caller":"binding/operation_volume_protection.go:106","msg":"succeed to init volume protection, pod: mysql-backup-mysql-0, spec: {\"highWatermark\":\"0\",\"volumes\":[]}"}
2023-10-11T06:01:38Z	INFO	Mysql	Initializing MySQL binding
2023-10-11T06:01:38Z	INFO	Redis	Initializing Redis binding
2023-10-11T06:01:38Z	INFO	Postgres	Initializing Postgres binding
2023-10-11T06:01:38Z	INFO	Mongo	Initializing MongoDB binding
2023-10-11T06:01:38Z	ERROR	HA	%s not set	{"characterType": "KB_SERVICE_CHARACTER_TYPE"}
github.com/apecloud/kubeblocks/lorry/highavailability.NewHa
	/src/lorry/highavailability/ha.go:55
main.main
	/src/cmd/lorry/main.go:98
runtime.main
	/usr/local/go/src/runtime/proc.go:267
2023-10-11T06:01:39Z	INFO	middleware	receive request	{"request": "/v1.0/bindings/custom?operation=checkRole"}
2023-10-11T06:01:39Z	INFO	middleware	No correspond builtin type, use the custom...
2023-10-11T06:01:39Z	INFO	Custom	action succeed	{"url": "http://127.0.0.1:36501/role?KB_RSM_LAST_STDOUT=", "output": ""}
2023-10-11T06:01:39Z	INFO	Custom	action succeed	{"url": "http://127.0.0.1:36502/role?KB_RSM_LAST_STDOUT=", "output": ""}
2023-10-11T06:01:39Z	INFO	Custom	operation called	{"operation": "checkRole", "result": {"event":"Success","operation":"checkRole","originalRole":"","role":""}}
2023-10-11T06:01:39Z	INFO	middleware	request routed	{"request": {"data":null,"metadata":{},"operation":"checkRole"}, "response": {"data":"eyJldmVudCI6IlN1Y2Nlc3MiLCJvcGVyYXRpb24iOiJjaGVja1JvbGUiLCJvcmlnaW5hbFJvbGUiOiIiLCJyb2xlIjoiIn0=","metadata":{"duration":"25.982576ms","end-time":"2023-10-11T06:01:39.514338967Z","operation":"checkRole","start-time":"2023-10-11T06:01:39.488356381Z"}}}
2023-10-11T06:01:39Z	INFO	middleware	response has no statusCodeHeader
2023-10-11T06:01:39Z	INFO	middleware	receive request	{"request": "/v1.0/bindings/custom?operation=checkRole"}
2023-10-11T06:01:39Z	INFO	middleware	No correspond builtin type, use the custom...
2023-10-11T06:01:39Z	INFO	Custom	action succeed	{"url": "http://127.0.0.1:36501/role?KB_RSM_LAST_STDOUT=", "output": ""}
2023-10-11T06:01:39Z	INFO	Custom	action succeed	{"url": "http://127.0.0.1:36502/role?KB_RSM_LAST_STDOUT=", "output": ""}
2023-10-11T06:01:39Z	INFO	Custom	operation called	{"operation": "checkRole", "result": {"event":"Success","operation":"checkRole","originalRole":"","role":""}}
2023-10-11T06:01:39Z	INFO	middleware	request routed	{"request": {"data":null,"metadata":{},"operation":"checkRole"}, "response": {"data":"eyJldmVudCI6IlN1Y2Nlc3MiLCJvcGVyYXRpb24iOiJjaGVja1JvbGUiLCJvcmlnaW5hbFJvbGUiOiIiLCJyb2xlIjoiIn0=","metadata":{"duration":"19.069973ms","end-time":"2023-10-11T06:01:39.54901545Z","operation":"checkRole","start-time":"2023-10-11T06:01:39.529945477Z"}}}
2023-10-11T06:01:39Z	INFO	middleware	response has no statusCodeHeader
2023-10-11T06:01:40Z	INFO	middleware	receive request	{"request": "/v1.0/bindings/custom?operation=checkRole"}
2023-10-11T06:01:40Z	INFO	middleware	No correspond builtin type, use the custom...
2023-10-11T06:01:40Z	INFO	Custom	action succeed	{"url": "http://127.0.0.1:36501/role?KB_RSM_LAST_STDOUT=", "output": ""}
2023-10-11T06:01:40Z	INFO	Custom	action succeed	{"url": "http://127.0.0.1:36502/role?KB_RSM_LAST_STDOUT=", "output": ""}
2023-10-11T06:01:40Z	INFO	Custom	operation called	{"operation": "checkRole", "result": {"event":"Success","operation":"checkRole","originalRole":"","role":""}}
2023-10-11T06:01:40Z	INFO	middleware	request routed	{"request": {"data":null,"metadata":{},"operation":"checkRole"}, "response": {"data":"eyJldmVudCI6IlN1Y2Nlc3MiLCJvcGVyYXRpb24iOiJjaGVja1JvbGUiLCJvcmlnaW5hbFJvbGUiOiIiLCJyb2xlIjoiIn0=","metadata":{"duration":"17.005872ms","end-time":"2023-10-11T06:01:40.546749909Z","operation":"checkRole","start-time":"2023-10-11T06:01:40.529744037Z"}}}
2023-10-11T06:01:40Z	INFO	middleware	response has no statusCodeHeader
2023-10-11T06:01:41Z	INFO	middleware	receive request	{"request": "/v1.0/bindings/custom?operation=checkRole"}
2023-10-11T06:01:41Z	INFO	middleware	No correspond builtin type, use the custom...
2023-10-11T06:01:41Z	INFO	Custom	action succeed	{"url": "http://127.0.0.1:36501/role?KB_RSM_LAST_STDOUT=", "output": ""}
2023-10-11T06:01:41Z	INFO	Custom	action succeed	{"url": "http://127.0.0.1:36502/role?KB_RSM_LAST_STDOUT=", "output": ""}
2023-10-11T06:01:41Z	INFO	Custom	operation called	{"operation": "checkRole", "result": {"event":"Success","operation":"checkRole","originalRole":"","role":""}}
2023-10-11T06:01:41Z	INFO	middleware	request routed	{"request": {"data":null,"metadata":{},"operation":"checkRole"}, "response": {"data":"eyJldmVudCI6IlN1Y2Nlc3MiLCJvcGVyYXRpb24iOiJjaGVja1JvbGUiLCJvcmlnaW5hbFJvbGUiOiIiLCJyb2xlIjoiIn0=","metadata":{"duration":"14.423245ms","end-time":"2023-10-11T06:01:41.544212179Z","operation":"checkRole","start-time":"2023-10-11T06:01:41.529788964Z"}}}
2023-10-11T06:01:41Z	INFO	middleware	response has no statusCodeHeader
2023-10-11T06:01:42Z	INFO	middleware	receive request	{"request": "/v1.0/bindings/custom?operation=checkRole"}
2023-10-11T06:01:42Z	INFO	middleware	No correspond builtin type, use the custom...
2023-10-11T06:01:42Z	INFO	Custom	action succeed	{"url": "http://127.0.0.1:36501/role?KB_RSM_LAST_STDOUT=", "output": ""}
2023-10-11T06:01:42Z	INFO	Custom	action succeed	{"url": "http://127.0.0.1:36502/role?KB_RSM_LAST_STDOUT=", "output": ""}
2023-10-11T06:01:42Z	INFO	Custom	operation called	{"operation": "checkRole", "result": {"event":"Success","operation":"checkRole","originalRole":"","role":""}}
2023-10-11T06:01:42Z	INFO	middleware	request routed	{"request": {"data":null,"metadata":{},"operation":"checkRole"}, "response": {"data":"eyJldmVudCI6IlN1Y2Nlc3MiLCJvcGVyYXRpb24iOiJjaGVja1JvbGUiLCJvcmlnaW5hbFJvbGUiOiIiLCJyb2xlIjoiIn0=","metadata":{"duration":"14.008643ms","end-time":"2023-10-11T06:01:42.543853291Z","operation":"checkRole","start-time":"2023-10-11T06:01:42.529844668Z"}}}
2023-10-11T06:01:42Z	INFO	middleware	response has no statusCodeHeader
2023-10-11T06:01:43Z	INFO	middleware	receive request	{"request": "/v1.0/bindings/custom?operation=checkRole"}
2023-10-11T06:01:43Z	INFO	middleware	No correspond builtin type, use the custom...
2023-10-11T06:01:43Z	INFO	Custom	action succeed	{"url": "http://127.0.0.1:36501/role?KB_RSM_LAST_STDOUT=", "output": ""}
2023-10-11T06:01:43Z	INFO	Custom	action succeed	{"url": "http://127.0.0.1:36502/role?KB_RSM_LAST_STDOUT=", "output": ""}
2023-10-11T06:01:43Z	INFO	Custom	operation called	{"operation": "checkRole", "result": {"event":"Success","operation":"checkRole","originalRole":"","role":""}}
2023-10-11T06:01:43Z	INFO	middleware	request routed	{"request": {"data":null,"metadata":{},"operation":"checkRole"}, "response": {"data":"eyJldmVudCI6IlN1Y2Nlc3MiLCJvcGVyYXRpb24iOiJjaGVja1JvbGUiLCJvcmlnaW5hbFJvbGUiOiIiLCJyb2xlIjoiIn0=","metadata":{"duration":"12.899481ms","end-time":"2023-10-11T06:01:43.542693756Z","operation":"checkRole","start-time":"2023-10-11T06:01:43.529794305Z"}}}
2023-10-11T06:01:43Z	INFO	middleware	response has no statusCodeHeader
2023-10-11T06:01:44Z	INFO	middleware	receive request	{"request": "/v1.0/bindings/custom?operation=checkRole"}
2023-10-11T06:01:44Z	INFO	middleware	No correspond builtin type, use the custom...
2023-10-11T06:01:44Z	INFO	Custom	action succeed	{"url": "http://127.0.0.1:36501/role?KB_RSM_LAST_STDOUT=", "output": ""}
2023-10-11T06:01:44Z	INFO	Custom	action succeed	{"url": "http://127.0.0.1:36502/role?KB_RSM_LAST_STDOUT=", "output": ""}
2023-10-11T06:01:44Z	INFO	Custom	operation called	{"operation": "checkRole", "result": {"event":"Success","operation":"checkRole","originalRole":"","role":""}}
2023-10-11T06:01:44Z	INFO	middleware	request routed	{"request": {"data":null,"metadata":{},"operation":"checkRole"}, "response": {"data":"eyJldmVudCI6IlN1Y2Nlc3MiLCJvcGVyYXRpb24iOiJjaGVja1JvbGUiLCJvcmlnaW5hbFJvbGUiOiIiLCJyb2xlIjoiIn0=","metadata":{"duration":"13.374195ms","end-time":"2023-10-11T06:01:44.542630073Z","operation":"checkRole","start-time":"2023-10-11T06:01:44.529255888Z"}}}
2023-10-11T06:01:44Z	INFO	middleware	response has no statusCodeHeader
2023-10-11T06:01:45Z	INFO	middleware	receive request	{"request": "/v1.0/bindings/custom?operation=checkRole"}
2023-10-11T06:01:45Z	INFO	middleware	No correspond builtin type, use the custom...
2023-10-11T06:01:45Z	INFO	Custom	action succeed	{"url": "http://127.0.0.1:36501/role?KB_RSM_LAST_STDOUT=", "output": ""}
2023-10-11T06:01:45Z	INFO	Custom	action succeed	{"url": "http://127.0.0.1:36502/role?KB_RSM_LAST_STDOUT=", "output": ""}
2023-10-11T06:01:45Z	INFO	Custom	operation called	{"operation": "checkRole", "result": {"event":"Success","operation":"checkRole","originalRole":"","role":""}}
2023-10-11T06:01:45Z	INFO	middleware	request routed	{"request": {"data":null,"metadata":{},"operation":"checkRole"}, "response": {"data":"eyJldmVudCI6IlN1Y2Nlc3MiLCJvcGVyYXRpb24iOiJjaGVja1JvbGUiLCJvcmlnaW5hbFJvbGUiOiIiLCJyb2xlIjoiIn0=","metadata":{"duration":"12.436138ms","end-time":"2023-10-11T06:01:45.541914341Z","operation":"checkRole","start-time":"2023-10-11T06:01:45.529478193Z"}}}
2023-10-11T06:01:45Z	INFO	middleware	response has no statusCodeHeader
2023-10-11T06:01:46Z	INFO	middleware	receive request	{"request": "/v1.0/bindings/custom?operation=checkRole"}
2023-10-11T06:01:46Z	INFO	middleware	No correspond builtin type, use the custom...
2023-10-11T06:01:46Z	INFO	Custom	action succeed	{"url": "http://127.0.0.1:36501/role?KB_RSM_LAST_STDOUT=", "output": ""}
2023-10-11T06:01:46Z	INFO	Custom	action succeed	{"url": "http://127.0.0.1:36502/role?KB_RSM_LAST_STDOUT=", "output": ""}
2023-10-11T06:01:46Z	INFO	Custom	operation called	{"operation": "checkRole", "result": {"event":"Success","operation":"checkRole","originalRole":"","role":""}}
2023-10-11T06:01:46Z	INFO	middleware	request routed	{"request": {"data":null,"metadata":{},"operation":"checkRole"}, "response": {"data":"eyJldmVudCI6IlN1Y2Nlc3MiLCJvcGVyYXRpb24iOiJjaGVja1JvbGUiLCJvcmlnaW5hbFJvbGUiOiIiLCJyb2xlIjoiIn0=","metadata":{"duration":"16.074274ms","end-time":"2023-10-11T06:01:46.54866792Z","operation":"checkRole","start-time":"2023-10-11T06:01:46.532593646Z"}}}
2023-10-11T06:01:46Z	INFO	middleware	response has no statusCodeHeader
2023-10-11T06:01:47Z	INFO	middleware	receive request	{"request": "/v1.0/bindings/custom?operation=checkRole"}
2023-10-11T06:01:47Z	INFO	middleware	No correspond builtin type, use the custom...
2023-10-11T06:01:47Z	INFO	Custom	action succeed	{"url": "http://127.0.0.1:36501/role?KB_RSM_LAST_STDOUT=", "output": ""}
2023-10-11T06:01:47Z	INFO	Custom	action succeed	{"url": "http://127.0.0.1:36502/role?KB_RSM_LAST_STDOUT=", "output": ""}
2023-10-11T06:01:47Z	INFO	Custom	operation called	{"operation": "checkRole", "result": {"event":"Success","operation":"checkRole","originalRole":"","role":""}}
2023-10-11T06:01:47Z	INFO	middleware	request routed	{"request": {"data":null,"metadata":{},"operation":"checkRole"}, "response": {"data":"eyJldmVudCI6IlN1Y2Nlc3MiLCJvcGVyYXRpb24iOiJjaGVja1JvbGUiLCJvcmlnaW5hbFJvbGUiOiIiLCJyb2xlIjoiIn0=","metadata":{"duration":"16.641001ms","end-time":"2023-10-11T06:01:47.546249347Z","operation":"checkRole","start-time":"2023-10-11T06:01:47.529608356Z"}}}

Expected behavior
mysql volume-snapshot backup restore success.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

  • OS: [e.g. iOS]
  • Browser [e.g. chrome, safari]
  • Version [e.g. 22]

Additional context
Add any other context about the problem here.
@JashBook JashBook added the kind/bug Something isn't working label Oct 11, 2023
@JashBook JashBook added this to the Release 0.7.0 milestone Oct 11, 2023
@JashBook JashBook added the severity/major Great chance user will encounter the same problem label Oct 11, 2023
@wangyelei wangyelei linked a pull request Oct 11, 2023 that will close this issue
fix: mysql xtrabackup failed and restore failed #5397
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@wangyelei wangyelei

Labels
bug kind/bug Something isn't working severity/major Great chance user will encounter the same problem
Projects
None yet
Milestone
Release 0.7.0
Development

Successfully merging a pull request may close this issue.

fix: mysql xtrabackup failed and restore failed apecloud/kubeblocks
3 participants
@wangyelei @JashBook @apecloud-bot