Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] cluster create failed on ACK /role-probe/agent error #5460

Closed
JashBook opened this issue Oct 16, 2023 · 2 comments
Closed

[BUG] cluster create failed on ACK /role-probe/agent error #5460

JashBook opened this issue Oct 16, 2023 · 2 comments
Assignees
@derecknowayback
Labels
bug kind/bug Something isn't working severity/major Great chance user will encounter the same problem
Milestone
Release 0.7.0

Comments

@JashBook
Copy link
Collaborator

Describe the bug
cluster create failed on ACK:
exec /role-probe/agent: exec format error.

kbcli version
Kubernetes: v1.26.3-aliyun.1
KubeBlocks: 0.7.0-beta.4
kbcli: 0.7.0-beta.4

To Reproduce
Steps to reproduce the behavior:

  1. create cluster
kbcli cluster create mysql mysql-test
  1. See error
kubectl get pod
NAME                                         READY   STATUS             RESTARTS         AGE
mysql-test-mysql-0                           6/8     CrashLoopBackOff   18 (3m38s ago)   26m

kubectl get cluster
NAME            CLUSTER-DEFINITION   VERSION           TERMINATION-POLICY   STATUS    AGE
mysql-test      apecloud-mysql       ac-mysql-8.0.30   Delete               Failed    27m

describe error pod

kubectl describe pod mysql-test-mysql-0
Name:             mysql-test-mysql-0
Namespace:        default
Priority:         0
Service Account:  kb-mysql-test
Node:             cn-zhangjiakou.10.0.86.119/10.0.86.119
Start Time:       Mon, 16 Oct 2023 14:15:04 +0800
Labels:           app.kubernetes.io/component=mysql
                  app.kubernetes.io/instance=mysql-test
                  app.kubernetes.io/managed-by=kubeblocks
                  app.kubernetes.io/name=apecloud-mysql
                  app.kubernetes.io/version=ac-mysql-8.0.30
                  apps.kubeblocks.io/component-name=mysql
                  apps.kubeblocks.io/workload-type=Consensus
                  controller-revision-hash=mysql-test-mysql-6dd76cd75f
                  kubeblocks.io/role=leader
                  rsm.workloads.kubeblocks.io/access-mode=ReadWrite
                  statefulset.kubernetes.io/pod-name=mysql-test-mysql-0
Annotations:      apps.kubeblocks.io/component-replicas: 1
                  apps.kubeblocks.io/last-role-snapshot-version: 2023-10-16T06:17:17.101706Z
                  k8s.aliyun.com/pod-ips: 10.0.86.186
Status:           Running
IP:               10.0.86.186
IPs:
  IP:           10.0.86.186
Controlled By:  StatefulSet/mysql-test-mysql
Init Containers:
  role-agent-installer:
    Container ID:  containerd://23eaee643478ae3d73bc04027de41f1586c34a4e020ce50bed6e6e98538a73ee
    Image:         msoap/shell2http:1.16.0
    Image ID:      docker.io/msoap/shell2http@sha256:a20bdde2f679de2cba6bf3d9f470489c7836d4d0d28232a2b295450809cd43ef
    Port:          <none>
    Host Port:     <none>
    Command:
      cp
      /app/shell2http
      /role-probe/agent
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Mon, 16 Oct 2023 14:15:17 +0800
      Finished:     Mon, 16 Oct 2023 14:15:17 +0800
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /role-probe from role-agent (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-lwmxl (ro)
Containers:
  mysql:
    Container ID:  containerd://6d157ee560612024c847edc7f24aaa9f792cf901256663504c2774ddbf80b6c2
    Image:         registry.cn-hangzhou.aliyuncs.com/apecloud/apecloud-mysql-server:8.0.30-5.beta2.20230830.g4ca1eb8.13
    Image ID:      registry.cn-hangzhou.aliyuncs.com/apecloud/apecloud-mysql-server@sha256:0f58f0cfac74cc502c7865c03ee53aead3bbb26fb9afb40bbef629fe47c0a3d6
    Ports:         3306/TCP, 13306/TCP
    Host Ports:    0/TCP, 0/TCP
    Command:
      /scripts/setup.sh
    State:          Running
      Started:      Mon, 16 Oct 2023 14:15:42 +0800
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     500m
      memory:  512Mi
    Requests:
      cpu:     500m
      memory:  512Mi
    Environment Variables from:
      mysql-test-mysql-env      ConfigMap  Optional: false
      mysql-test-mysql-rsm-env  ConfigMap  Optional: false
    Environment:
      KB_POD_NAME:               mysql-test-mysql-0 (v1:metadata.name)
      KB_POD_UID:                 (v1:metadata.uid)
      KB_NAMESPACE:              default (v1:metadata.namespace)
      KB_SA_NAME:                 (v1:spec.serviceAccountName)
      KB_NODENAME:                (v1:spec.nodeName)
      KB_HOST_IP:                 (v1:status.hostIP)
      KB_POD_IP:                  (v1:status.podIP)
      KB_POD_IPS:                 (v1:status.podIPs)
      KB_HOSTIP:                  (v1:status.hostIP)
      KB_PODIP:                   (v1:status.podIP)
      KB_PODIPS:                  (v1:status.podIPs)
      KB_CLUSTER_NAME:           mysql-test
      KB_COMP_NAME:              mysql
      KB_CLUSTER_COMP_NAME:      mysql-test-mysql
      KB_CLUSTER_UID_POSTFIX_8:  519edc07
      KB_POD_FQDN:               $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
      MYSQL_ROOT_HOST:           %
      MYSQL_ROOT_USER:           <set to the key 'username' in secret 'mysql-test-conn-credential'>  Optional: false
      MYSQL_ROOT_PASSWORD:       <set to the key 'password' in secret 'mysql-test-conn-credential'>  Optional: false
      MYSQL_DATABASE:            mydb
      MYSQL_USER:                u1
      MYSQL_PASSWORD:            u1
      CLUSTER_ID:                1
      CLUSTER_START_INDEX:       1
      REPLICATION_USER:          replicator
      REPLICATION_PASSWORD:      
      MYSQL_TEMPLATE_CONFIG:     
      MYSQL_CUSTOM_CONFIG:       
      MYSQL_DYNAMIC_CONFIG:      
      KB_EMBEDDED_WESQL:         1
    Mounts:
      /data/mysql from data (rw)
      /etc/annotations from annotations (rw)
      /opt/mysql from mysql-config (rw)
      /scripts from scripts (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-lwmxl (ro)
  metrics:
    Container ID:  containerd://62d2663f2b11dcf5c52b2dc9120a5ef9750cb2adb82d98058eb7cb830867205a
    Image:         registry.cn-hangzhou.aliyuncs.com/apecloud/agamotto:0.1.2-beta.1
    Image ID:      registry.cn-hangzhou.aliyuncs.com/apecloud/agamotto@sha256:cbab349b90490807a8d5039bf01bc7e37334f20c98c7dd75bc7fc4cf9e5b10ee
    Port:          9104/TCP
    Host Port:     0/TCP
    Command:
      /scripts/agamotto.sh
    State:          Running
      Started:      Mon, 16 Oct 2023 14:15:53 +0800
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     0
      memory:  0
    Requests:
      cpu:     0
      memory:  0
    Environment Variables from:
      mysql-test-mysql-env      ConfigMap  Optional: false
      mysql-test-mysql-rsm-env  ConfigMap  Optional: false
    Environment:
      KB_POD_NAME:               mysql-test-mysql-0 (v1:metadata.name)
      KB_POD_UID:                 (v1:metadata.uid)
      KB_NAMESPACE:              default (v1:metadata.namespace)
      KB_SA_NAME:                 (v1:spec.serviceAccountName)
      KB_NODENAME:                (v1:spec.nodeName)
      KB_HOST_IP:                 (v1:status.hostIP)
      KB_POD_IP:                  (v1:status.podIP)
      KB_POD_IPS:                 (v1:status.podIPs)
      KB_HOSTIP:                  (v1:status.hostIP)
      KB_PODIP:                   (v1:status.podIP)
      KB_PODIPS:                  (v1:status.podIPs)
      KB_CLUSTER_NAME:           mysql-test
      KB_COMP_NAME:              mysql
      KB_CLUSTER_COMP_NAME:      mysql-test-mysql
      KB_CLUSTER_UID_POSTFIX_8:  519edc07
      KB_POD_FQDN:               $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
      DB_TYPE:                   MySQL
      ENDPOINT:                  localhost:3306
      MYSQL_USER:                <set to the key 'username' in secret 'mysql-test-conn-credential'>  Optional: false
      MYSQL_PASSWORD:            <set to the key 'password' in secret 'mysql-test-conn-credential'>  Optional: false
    Mounts:
      /data/mysql from data (rw)
      /opt/agamotto from agamotto-configuration (rw)
      /scripts from scripts (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-lwmxl (ro)
  vttablet:
    Container ID:  containerd://76f24abbfc8acc4f32bb84c11e95b6a53297691643725992b21adb91f23ad815
    Image:         registry.cn-hangzhou.aliyuncs.com/apecloud/apecloud-mysql-scale:0.1.1
    Image ID:      registry.cn-hangzhou.aliyuncs.com/apecloud/apecloud-mysql-scale@sha256:d391dde6e2f3f6a6056a4c41fd7e255f71ae94618b08e6afdd9527f6892a761b
    Ports:         15100/TCP, 16100/TCP
    Host Ports:    0/TCP, 0/TCP
    Command:
      /scripts/vttablet.sh
    State:          Running
      Started:      Mon, 16 Oct 2023 14:16:50 +0800
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     0
      memory:  0
    Requests:
      cpu:     0
      memory:  0
    Environment Variables from:
      mysql-test-mysql-env      ConfigMap  Optional: false
      mysql-test-mysql-rsm-env  ConfigMap  Optional: false
    Environment:
      KB_POD_NAME:               mysql-test-mysql-0 (v1:metadata.name)
      KB_POD_UID:                 (v1:metadata.uid)
      KB_NAMESPACE:              default (v1:metadata.namespace)
      KB_SA_NAME:                 (v1:spec.serviceAccountName)
      KB_NODENAME:                (v1:spec.nodeName)
      KB_HOST_IP:                 (v1:status.hostIP)
      KB_POD_IP:                  (v1:status.podIP)
      KB_POD_IPS:                 (v1:status.podIPs)
      KB_HOSTIP:                  (v1:status.hostIP)
      KB_PODIP:                   (v1:status.podIP)
      KB_PODIPS:                  (v1:status.podIPs)
      KB_CLUSTER_NAME:           mysql-test
      KB_COMP_NAME:              mysql
      KB_CLUSTER_COMP_NAME:      mysql-test-mysql
      KB_CLUSTER_UID_POSTFIX_8:  519edc07
      KB_POD_FQDN:               $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
      CELL:                      zone1
      ETCD_SERVER:               $(KB_CLUSTER_NAME)-vtcontroller-headless
      ETCD_PORT:                 2379
      TOPOLOGY_FLAGS:            --topo_implementation etcd2 --topo_global_server_address $(ETCD_SERVER):$(ETCD_PORT) --topo_global_root /vitess/global
      VTTABLET_PORT:             15100
      VTTABLET_GRPC_PORT:        16100
      VTCTLD_HOST:               $(KB_CLUSTER_NAME)-vtcontroller-headless
      VTCTLD_WEB_PORT:           15000
      MYSQL_ROOT_USER:           <set to the key 'username' in secret 'mysql-test-conn-credential'>  Optional: false
      MYSQL_ROOT_PASSWORD:       <set to the key 'password' in secret 'mysql-test-conn-credential'>  Optional: false
    Mounts:
      /conf from mysql-scale-config (rw)
      /scripts from scripts (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-lwmxl (ro)
      /vtdataroot from data (rw)
  kb-checkrole:
    Container ID:  containerd://7fc04cb451f76a390be06c40408b515c72e2fe330d3726155a747b9072cab778
    Image:         registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools:0.7.0-beta.4
    Image ID:      registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools@sha256:25c4004565f7b498395852935118b897efdfbbc6eebddf3d1662c4eb6ba79514
    Port:          3501/TCP
    Host Port:     0/TCP
    Command:
      lorry
      --port
      3501
    State:          Running
      Started:      Mon, 16 Oct 2023 14:16:50 +0800
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     0
      memory:  0
    Requests:
      cpu:      0
      memory:   0
    Readiness:  http-get http://:3501/v1.0/bindings/mysql%3Foperation=checkRole&workloadType=Consensus delay=0s timeout=1s period=1s #success=1 #failure=2
    Startup:    tcp-socket :3501 delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment Variables from:
      mysql-test-mysql-env      ConfigMap  Optional: false
      mysql-test-mysql-rsm-env  ConfigMap  Optional: false
    Environment:
      KB_POD_NAME:                mysql-test-mysql-0 (v1:metadata.name)
      KB_POD_UID:                  (v1:metadata.uid)
      KB_NAMESPACE:               default (v1:metadata.namespace)
      KB_SA_NAME:                  (v1:spec.serviceAccountName)
      KB_NODENAME:                 (v1:spec.nodeName)
      KB_HOST_IP:                  (v1:status.hostIP)
      KB_POD_IP:                   (v1:status.podIP)
      KB_POD_IPS:                  (v1:status.podIPs)
      KB_HOSTIP:                   (v1:status.hostIP)
      KB_PODIP:                    (v1:status.podIP)
      KB_PODIPS:                   (v1:status.podIPs)
      KB_CLUSTER_NAME:            mysql-test
      KB_COMP_NAME:               mysql
      KB_CLUSTER_COMP_NAME:       mysql-test-mysql
      KB_CLUSTER_UID_POSTFIX_8:   519edc07
      KB_POD_FQDN:                $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
      KB_SERVICE_USER:            <set to the key 'username' in secret 'mysql-test-conn-credential'>  Optional: false
      KB_SERVICE_PASSWORD:        <set to the key 'password' in secret 'mysql-test-conn-credential'>  Optional: false
      KB_SERVICE_PORT:            3306
      KB_DATA_PATH:               /data/mysql
      KB_SERVICE_CHARACTER_TYPE:  mysql
      KB_WORKLOAD_TYPE:           Consensus
      KB_SERVICE_USER:            <set to the key 'username' in secret 'mysql-test-conn-credential'>  Optional: false
      KB_SERVICE_PASSWORD:        <set to the key 'password' in secret 'mysql-test-conn-credential'>  Optional: false
    Mounts:
      /data/mysql from data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-lwmxl (ro)
  config-manager:
    Container ID:  containerd://742c0d7cc9d7a2625b53dafe3bead74a220424875e010e04dafe406fba326867
    Image:         registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools:0.7.0-beta.4
    Image ID:      registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools@sha256:25c4004565f7b498395852935118b897efdfbbc6eebddf3d1662c4eb6ba79514
    Port:          <none>
    Host Port:     <none>
    Command:
      env
    Args:
      PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:$(TOOLS_PATH)
      /bin/reloader
      --log-level
      info
      --operator-update-enable
      --tcp
      9901
      --config
      /opt/config-manager/config-manager.yaml
    State:          Running
      Started:      Mon, 16 Oct 2023 14:16:50 +0800
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     0
      memory:  0
    Requests:
      cpu:     0
      memory:  0
    Environment Variables from:
      mysql-test-mysql-env      ConfigMap  Optional: false
      mysql-test-mysql-rsm-env  ConfigMap  Optional: false
    Environment:
      KB_POD_NAME:               mysql-test-mysql-0 (v1:metadata.name)
      KB_POD_UID:                 (v1:metadata.uid)
      KB_NAMESPACE:              default (v1:metadata.namespace)
      KB_SA_NAME:                 (v1:spec.serviceAccountName)
      KB_NODENAME:                (v1:spec.nodeName)
      KB_HOST_IP:                 (v1:status.hostIP)
      KB_POD_IP:                  (v1:status.podIP)
      KB_POD_IPS:                 (v1:status.podIPs)
      KB_HOSTIP:                  (v1:status.hostIP)
      KB_PODIP:                   (v1:status.podIP)
      KB_PODIPS:                  (v1:status.podIPs)
      KB_CLUSTER_NAME:           mysql-test
      KB_COMP_NAME:              mysql
      KB_CLUSTER_COMP_NAME:      mysql-test-mysql
      KB_CLUSTER_UID_POSTFIX_8:  519edc07
      KB_POD_FQDN:               $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
      CONFIG_MANAGER_POD_IP:      (v1:status.podIP)
      DB_TYPE:                   mysql
      MYSQL_USER:                <set to the key 'username' in secret 'mysql-test-conn-credential'>  Optional: false
      MYSQL_PASSWORD:            <set to the key 'password' in secret 'mysql-test-conn-credential'>  Optional: false
      DATA_SOURCE_NAME:          $(MYSQL_USER):$(MYSQL_PASSWORD)@(localhost:3306)/
      TOOLS_PATH:                /opt/kb-tools/reload/mysql-consensusset-config:/opt/config-manager
    Mounts:
      /conf from mysql-scale-config (rw)
      /opt/config-manager from config-manager-config (rw)
      /opt/kb-tools/reload/mysql-consensusset-config from cm-script-mysql-consensusset-config (rw)
      /opt/mysql from mysql-config (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-lwmxl (ro)
  action-0:
    Container ID:  containerd://fa9ed4f5930053b6220e42c2e133ee0937662ed1aa04942b4adb05d20521f41f
    Image:         arey/mysql-client:latest
    Image ID:      docker.io/arey/mysql-client@sha256:3e80958e3c89a3d75971dc62a87876a5f35578a2adf93016b2d8cfb1fe10dfd9
    Port:          <none>
    Host Port:     <none>
    Command:
      /role-probe/agent
      -port
      36501
      -export-all-vars
      -form
      /role
      mysql -h127.0.0.1 -P3306 -u$KB_RSM_USERNAME -p$KB_RSM_PASSWORD -N -B -e "select role from information_schema.wesql_cluster_local" | xargs echo -n
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Mon, 16 Oct 2023 14:38:24 +0800
      Finished:     Mon, 16 Oct 2023 14:38:24 +0800
    Ready:          False
    Restart Count:  9
    Environment:
      KB_RSM_USERNAME:  <set to the key 'username' in secret 'mysql-test-conn-credential'>  Optional: false
      KB_RSM_PASSWORD:  <set to the key 'password' in secret 'mysql-test-conn-credential'>  Optional: false
    Mounts:
      /role-probe from role-agent (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-lwmxl (ro)
  action-1:
    Container ID:  containerd://df5f1149ac558c6cd20f267d04dd9b825bf849a6a3c358770d6c4b40bf54e8d0
    Image:         busybox:latest
    Image ID:      docker.io/library/busybox@sha256:3fbc632167424a6d997e74f52b878d7cc478225cffac6bc977eedfe51c7f4e79
    Port:          <none>
    Host Port:     <none>
    Command:
      /role-probe/agent
      -port
      36502
      -export-all-vars
      -form
      /role
      echo $v_KB_RSM_LAST_STDOUT | tr '[:upper:]' '[:lower:]' | xargs echo -n
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Mon, 16 Oct 2023 14:38:24 +0800
      Finished:     Mon, 16 Oct 2023 14:38:24 +0800
    Ready:          False
    Restart Count:  9
    Environment:
      KB_RSM_USERNAME:  <set to the key 'username' in secret 'mysql-test-conn-credential'>  Optional: false
      KB_RSM_PASSWORD:  <set to the key 'password' in secret 'mysql-test-conn-credential'>  Optional: false
    Mounts:
      /role-probe from role-agent (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-lwmxl (ro)
  kb-role-probe:
    Container ID:  containerd://6f77a73c2e0f274dfcf4aacfc83dd33924a5bdb2f53b4dce20627c0dec42a899
    Image:         apecloud/kubeblocks-tools:latest
    Image ID:      docker.io/apecloud/kubeblocks-tools@sha256:71889f010d8226383dd59fe5bb4c3aca532df5603879ba0befe6775f443c1b65
    Ports:         7373/TCP, 50101/TCP
    Host Ports:    0/TCP, 0/TCP
    Command:
      lorry
      --port
      7373
      --grpcport
      50101
    State:          Running
      Started:      Mon, 16 Oct 2023 14:17:12 +0800
    Ready:          True
    Restart Count:  0
    Readiness:      exec [/bin/grpc_health_probe -addr=:50101] delay=0s timeout=1s period=1s #success=1 #failure=2
    Environment:
      KB_RSM_USERNAME:               <set to the key 'username' in secret 'mysql-test-conn-credential'>  Optional: false
      KB_RSM_PASSWORD:               <set to the key 'password' in secret 'mysql-test-conn-credential'>  Optional: false
      KB_RSM_ACTION_SVC_LIST:        [36501,36502]
      KB_SERVICE_USER:               <set to the key 'username' in secret 'mysql-test-conn-credential'>  Optional: false
      KB_SERVICE_PASSWORD:           <set to the key 'password' in secret 'mysql-test-conn-credential'>  Optional: false
      KB_RSM_SERVICE_PORT:           3306
      KB_SERVICE_PORT:               3306
      KB_RSM_ROLE_UPDATE_MECHANISM:  DirectAPIServerEventUpdate
      KB_RSM_ROLE_PROBE_TIMEOUT:     1
      KB_POD_NAME:                   mysql-test-mysql-0 (v1:metadata.name)
      KB_NAMESPACE:                  default (v1:metadata.namespace)
      KB_POD_UID:                     (v1:metadata.uid)
      KB_NODENAME:                    (v1:spec.nodeName)
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-lwmxl (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  data-mysql-test-mysql-0
    ReadOnly:   false
  annotations:
    Type:  DownwardAPI (a volume populated by information about the pod)
    Items:
      metadata.annotations['cs.apps.kubeblocks.io/leader'] -> leader
      metadata.annotations['apps.kubeblocks.io/component-replicas'] -> component-replicas
  agamotto-configuration:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      mysql-test-mysql-agamotto-configuration
    Optional:  false
  scripts:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      mysql-test-mysql-apecloud-mysql-scripts
    Optional:  false
  mysql-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      mysql-test-mysql-mysql-consensusset-config
    Optional:  false
  mysql-scale-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      mysql-test-mysql-vttablet-config
    Optional:  false
  cm-script-mysql-consensusset-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      sidecar-mysql-reload-script-mysql-test
    Optional:  false
  config-manager-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      sidecar-mysql-test-mysql-config-manager-config
    Optional:  false
  role-agent:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  kube-api-access-lwmxl:
    Type:                     Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:   3607
    ConfigMapName:            kube-root-ca.crt
    ConfigMapOptional:        <nil>
    DownwardAPI:              true
QoS Class:                    Burstable
Node-Selectors:               <none>
Tolerations:                  kb-data=true:NoSchedule
                              node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                              node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Topology Spread Constraints:  kubernetes.io/hostname:DoNotSchedule when max skew 1 is exceeded for selector app.kubernetes.io/instance=mysql-test,apps.kubeblocks.io/component-name=mysql
Events:
  Type     Reason                  Age                   From                     Message
  ----     ------                  ----                  ----                     -------
  Normal   Scheduled               28m                   default-scheduler        Successfully assigned default/mysql-test-mysql-0 to cn-zhangjiakou.10.0.86.119
  Normal   SuccessfulAttachVolume  28m                   attachdetach-controller  AttachVolume.Attach succeeded for volume "d-8vb9hu2k9ltka8wmih08"
  Normal   AllocIPSucceed          28m                   terway-daemon            Alloc IP 10.0.86.186/16
  Normal   Pulled                  28m                   kubelet                  Container image "msoap/shell2http:1.16.0" already present on machine
  Normal   Created                 28m                   kubelet                  Created container role-agent-installer
  Normal   Started                 28m                   kubelet                  Started container role-agent-installer
  Normal   Pulling                 28m                   kubelet                  Pulling image "registry.cn-hangzhou.aliyuncs.com/apecloud/apecloud-mysql-server:8.0.30-5.beta2.20230830.g4ca1eb8.13"
  Normal   Pulled                  27m                   kubelet                  Successfully pulled image "registry.cn-hangzhou.aliyuncs.com/apecloud/apecloud-mysql-server:8.0.30-5.beta2.20230830.g4ca1eb8.13" in 24.172757759s (24.172770679s including waiting)
  Normal   Created                 27m                   kubelet                  Created container mysql
  Normal   Started                 27m                   kubelet                  Started container mysql
  Normal   Pulling                 27m                   kubelet                  Pulling image "registry.cn-hangzhou.aliyuncs.com/apecloud/agamotto:0.1.2-beta.1"
  Normal   Pulling                 27m                   kubelet                  Pulling image "registry.cn-hangzhou.aliyuncs.com/apecloud/apecloud-mysql-scale:0.1.1"
  Normal   Created                 27m                   kubelet                  Created container metrics
  Normal   Started                 27m                   kubelet                  Started container metrics
  Normal   Pulled                  27m                   kubelet                  Successfully pulled image "registry.cn-hangzhou.aliyuncs.com/apecloud/agamotto:0.1.2-beta.1" in 11.62758232s (11.627590699s including waiting)
  Normal   Started                 26m                   kubelet                  Started container kb-checkrole
  Normal   Created                 26m                   kubelet                  Created container config-manager
  Normal   Started                 26m                   kubelet                  Started container vttablet
  Normal   Pulled                  26m                   kubelet                  Container image "registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools:0.7.0-beta.4" already present on machine
  Normal   Created                 26m                   kubelet                  Created container kb-checkrole
  Normal   Pulled                  26m                   kubelet                  Successfully pulled image "registry.cn-hangzhou.aliyuncs.com/apecloud/apecloud-mysql-scale:0.1.1" in 56.670968818s (56.670980778s including waiting)
  Normal   Pulled                  26m                   kubelet                  Container image "registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools:0.7.0-beta.4" already present on machine
  Normal   Created                 26m                   kubelet                  Created container vttablet
  Normal   Started                 26m                   kubelet                  Started container config-manager
  Normal   Pulling                 26m                   kubelet                  Pulling image "arey/mysql-client:latest"
  Normal   Pulled                  26m                   kubelet                  Successfully pulled image "arey/mysql-client:latest" in 11.045307214s (11.045319474s including waiting)
  Normal   Created                 26m                   kubelet                  Created container action-0
  Normal   Started                 26m                   kubelet                  Started container action-0
  Normal   checkRole               26m                   sqlchannel               {"event":"Failed","message":"Post \"http://127.0.0.1:36501/role?KB_RSM_LAST_STDOUT=\": dial tcp 127.0.0.1:36501: connect: connection refused","operation":"checkRole","originalRole":""}
  Normal   checkRole               26m                   sqlchannel               {"event":"Success","operation":"checkRole","originalRole":"","role":"Leader"}
  Warning  BackOff                 3m9s (x120 over 26m)  kubelet                  Back-off restarting failed container action-0 in pod mysql-test-mysql-0_default(ed128a08-20f5-476d-9e2c-3ae905b695b5)
➜  ~

logs error pod

➜  ~ kubectl logs mysql-test-mysql-0 action-0
exec /role-probe/agent: exec format error

Expected behavior
cluster create success on ACK

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

  • OS: [e.g. iOS]
  • Browser [e.g. chrome, safari]
  • Version [e.g. 22]

Additional context
Add any other context about the problem here.
@JashBook JashBook added kind/bug Something isn't working severity/major Great chance user will encounter the same problem labels Oct 16, 2023
@JashBook JashBook added this to the Release 0.7.0 milestone Oct 16, 2023
@derecknowayback
Copy link
Contributor

derecknowayback commented Oct 16, 2023 •
edited
Loading

@JashBook What is the os of aliyun? It seems similar to #5392

@JashBook
Copy link
Collaborator Author

JashBook commented Oct 16, 2023 •
edited
Loading

@JashBook What is the os of aliyun? It seems similar to #5392

Aliyun Linux3 Arm64

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@derecknowayback derecknowayback

Labels
bug kind/bug Something isn't working severity/major Great chance user will encounter the same problem
Projects
None yet
Milestone
Release 0.7.0
Development

No branches or pull requests
4 participants
@xuriwuyun @derecknowayback @JashBook @apecloud-bot