Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] redis pod CrashLoopBackOff after kill 1 #8279

Open
JashBook opened this issue Oct 15, 2024 · 1 comment
Open

[BUG] redis pod CrashLoopBackOff after kill 1 #8279

JashBook opened this issue Oct 15, 2024 · 1 comment
Assignees
@Y-Rookie
Labels
kind/bug Something isn't working Stale
Milestone
Release 0.9.3

Comments

@JashBook
Copy link
Collaborator

JashBook commented Oct 15, 2024 •
edited
Loading

Describe the bug

kbcli version
Kubernetes: v1.28.13-eks-a737599
KubeBlocks: 0.9.1
kbcli: 0.9.1

To Reproduce
Steps to reproduce the behavior:

  1. install redis addon
helm upgrade --install --namespace kb-system kb-addon-redis apecloud-addons/redis --version 0.9.1 --set image.registry=apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com --set images.registry=apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com
  1. create cluster
apiVersion: apps.kubeblocks.io/v1alpha1
kind: Cluster
metadata:
  name: redis-jwrrzl
  namespace: default
spec:
  terminationPolicy: WipeOut
  componentSpecs:
    - name: redis
      componentDef: redis-7
      replicas: 2
      resources:
        requests:
          cpu: 100m
          memory: 0.5Gi
        limits:
          cpu: 100m
          memory: 0.5Gi
      switchPolicy:
        type: Noop
      volumeClaimTemplates:
        - name: data
          spec:
            storageClassName:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 20Gi
    - name: redis-sentinel
      componentDef: redis-sentinel-7
      replicas: 3
      resources:
        requests:
          cpu: 100m
          memory: 0.5Gi
        limits:
          cpu: 100m
          memory: 0.5Gi
      volumeClaimTemplates:
        - name: data
          spec:
            storageClassName:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 20Gi
  1. see cluster
kbcli cluster list-instances redis-jwrrzl --namespace default `
    
NAME                            NAMESPACE   CLUSTER        COMPONENT        STATUS    ROLE        ACCESSMODE   AZ              CPU(REQUEST/LIMIT)   MEMORY(REQUEST/LIMIT)           STORAGE     NODE                      CREATED-TIME                 
redis-jwrrzl-redis-0            default     redis-jwrrzl   redis            Running   secondary   <none>       cn-shanghai-b   200m / 200m          644245094400m / 644245094400m   data:22Gi   172.16.0.40/172.16.0.40   Oct 15,2024 17:33 UTC+0800   
redis-jwrrzl-redis-1            default     redis-jwrrzl   redis            Running   primary     <none>       cn-shanghai-b   200m / 200m          644245094400m / 644245094400m   data:22Gi   172.16.0.72/172.16.0.72   Oct 15,2024 17:32 UTC+0800   
redis-jwrrzl-redis-sentinel-0   default     redis-jwrrzl   redis-sentinel   Running   <none>      <none>       cn-shanghai-b   100m / 100m          512Mi / 512Mi                   data:20Gi   172.16.0.2/172.16.0.2     Oct 15,2024 17:33 UTC+0800   
redis-jwrrzl-redis-sentinel-1   default     redis-jwrrzl   redis-sentinel   Running   <none>      <none>       cn-shanghai-b   100m / 100m          512Mi / 512Mi                   data:20Gi   172.16.0.40/172.16.0.40   Oct 15,2024 17:33 UTC+0800   
redis-jwrrzl-redis-sentinel-2   default     redis-jwrrzl   redis-sentinel   Running   <none>      <none>       cn-shanghai-b   100m / 100m          512Mi / 512Mi                   data:20Gi   172.16.0.72/172.16.0.72   Oct 15,2024 17:32 UTC+0800
  1. kill 1
kubectl exec -it redis-jwrrzl-redis-1 bash
kill 1
  1. See error
kubectl get pod -l app.kubernetes.io/instance=redis-jwrrzl                            
NAME                            READY   STATUS             RESTARTS      AGE
redis-jwrrzl-redis-0            3/3     Running            0             12m
redis-jwrrzl-redis-1            2/3     CrashLoopBackOff   7 (18s ago)   12m
redis-jwrrzl-redis-sentinel-0   2/2     Running            0             11m
redis-jwrrzl-redis-sentinel-1   2/2     Running            0             12m
redis-jwrrzl-redis-sentinel-2   2/2     Running            0             12m
➜  ~ 
➜  ~ kubectl get cluster redis-jwrrzl
NAME           CLUSTER-DEFINITION   VERSION   TERMINATION-POLICY   STATUS     AGE
redis-jwrrzl                                  WipeOut              Abnormal   22m
➜  ~ 
➜  ~ kubectl get cmp -l app.kubernetes.io/instance=redis-jwrrzl       
NAME                          DEFINITION         SERVICE-VERSION   STATUS     AGE
redis-jwrrzl-redis            redis-7            7.2.4             Abnormal   23m
redis-jwrrzl-redis-sentinel   redis-sentinel-7   7.2.4             Running    23m

describe pod

kubectl describe pod redis-jwrrzl-redis-1 
Name:             redis-jwrrzl-redis-1
Namespace:        default
Priority:         0
Service Account:  kb-redis-jwrrzl
Node:             172.16.0.72/172.16.0.72
Start Time:       Tue, 15 Oct 2024 17:32:38 +0800
Labels:           app.kubernetes.io/component=redis-7
                  app.kubernetes.io/instance=redis-jwrrzl
                  app.kubernetes.io/managed-by=kubeblocks
                  app.kubernetes.io/name=redis-7
                  app.kubernetes.io/version=redis-7
                  apps.kubeblocks.io/cluster-uid=feb40800-d600-4844-9a8d-1961e36dce19
                  apps.kubeblocks.io/component-name=redis
                  apps.kubeblocks.io/pod-name=redis-jwrrzl-redis-1
                  componentdefinition.kubeblocks.io/name=redis-7
                  controller-revision-hash=79fb5498d
                  kubeblocks.io/role=primary
                  workloads.kubeblocks.io/access-mode=ReadWrite
                  workloads.kubeblocks.io/instance=redis-jwrrzl-redis
                  workloads.kubeblocks.io/managed-by=InstanceSet
Annotations:      apps.kubeblocks.io/component-replicas: 2
                  apps.kubeblocks.io/last-role-snapshot-version: 1728984792862705
                  config.kubeblocks.io/restart-redis-replication-config: 85ddfc84cf
                  kubeblocks.io/restart: 2024-10-15T09:32:38Z
                  vke.volcengine.com/cello-pod-evict-policy: allow
Status:           Running
IP:               172.16.0.74
IPs:
  IP:           172.16.0.74
Controlled By:  InstanceSet/redis-jwrrzl-redis
Init Containers:
  init-lorry:
    Container ID:  containerd://1f0935cce0ac758cf92204ae5418829ad957e8908d70183a2f801dad61a0ee60
    Image:         apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/kubeblocks-tools:0.9.1
    Image ID:      apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/kubeblocks-tools@sha256:6afd857c62229b58eec81cfd51deb27cedd03630b471fad57a46c3d7da9662fd
    Port:          <none>
    Host Port:     <none>
    Command:
      cp
      -r
      /bin/lorry
      /config
      /bin/curl
      /kubeblocks/
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Tue, 15 Oct 2024 17:32:53 +0800
      Finished:     Tue, 15 Oct 2024 17:32:53 +0800
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     0
      memory:  0
    Requests:
      cpu:     0
      memory:  0
    Environment Variables from:
      redis-jwrrzl-redis-env  ConfigMap  Optional: false
    Environment:
      REDIS_DEFAULT_USER:      <set to the key 'username' in secret 'redis-jwrrzl-redis-account-default'>           Optional: false
      REDIS_DEFAULT_PASSWORD:  <set to the key 'password' in secret 'redis-jwrrzl-redis-account-default'>           Optional: false
      SENTINEL_USER:           <set to the key 'username' in secret 'redis-jwrrzl-redis-sentinel-account-default'>  Optional: false
      SENTINEL_PASSWORD:       <set to the key 'password' in secret 'redis-jwrrzl-redis-sentinel-account-default'>  Optional: false
      KB_POD_NAME:             redis-jwrrzl-redis-1 (v1:metadata.name)
      KB_POD_UID:               (v1:metadata.uid)
      KB_NAMESPACE:            default (v1:metadata.namespace)
      KB_SA_NAME:               (v1:spec.serviceAccountName)
      KB_NODENAME:              (v1:spec.nodeName)
      KB_HOST_IP:               (v1:status.hostIP)
      KB_POD_IP:                (v1:status.podIP)
      KB_POD_IPS:               (v1:status.podIPs)
      KB_HOSTIP:                (v1:status.hostIP)
      KB_PODIP:                 (v1:status.podIP)
      KB_PODIPS:                (v1:status.podIPs)
      KB_POD_FQDN:             $(KB_POD_NAME).redis-jwrrzl-redis-headless.$(KB_NAMESPACE).svc
    Mounts:
      /kubeblocks from kubeblocks (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-5bmdc (ro)
Containers:
  redis:
    Container ID:  containerd://694a8495c465e29d671ec3f51916c463315bacfa17c169a0869b6f58b53ffad8
    Image:         apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/redis-stack-server:7.2.0-v10
    Image ID:      apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/redis-stack-server@sha256:f55712f253ffafedd9f403cee24bc6644a26e058d48a65037bc64b6d31a86349
    Port:          6379/TCP
    Host Port:     0/TCP
    Command:
      /scripts/redis-start.sh
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Tue, 15 Oct 2024 17:45:02 +0800
      Finished:     Tue, 15 Oct 2024 17:45:03 +0800
    Ready:          False
    Restart Count:  7
    Limits:
      cpu:                        200m
      memory:                     644245094400m
      vke.volcengine.com/eni-ip:  1
    Requests:
      cpu:                        200m
      memory:                     644245094400m
      vke.volcengine.com/eni-ip:  1
    Readiness:                    exec [sh -c /scripts/redis-ping.sh 1] delay=10s timeout=1s period=5s #success=1 #failure=5
    Environment Variables from:
      redis-jwrrzl-redis-env      ConfigMap  Optional: false
      redis-jwrrzl-redis-rsm-env  ConfigMap  Optional: false
    Environment:
      REDIS_DEFAULT_USER:       <set to the key 'username' in secret 'redis-jwrrzl-redis-account-default'>           Optional: false
      REDIS_DEFAULT_PASSWORD:   <set to the key 'password' in secret 'redis-jwrrzl-redis-account-default'>           Optional: false
      SENTINEL_USER:            <set to the key 'username' in secret 'redis-jwrrzl-redis-sentinel-account-default'>  Optional: false
      SENTINEL_PASSWORD:        <set to the key 'password' in secret 'redis-jwrrzl-redis-sentinel-account-default'>  Optional: false
      KB_POD_NAME:              redis-jwrrzl-redis-1 (v1:metadata.name)
      KB_POD_UID:                (v1:metadata.uid)
      KB_NAMESPACE:             default (v1:metadata.namespace)
      KB_SA_NAME:                (v1:spec.serviceAccountName)
      KB_NODENAME:               (v1:spec.nodeName)
      KB_HOST_IP:                (v1:status.hostIP)
      KB_POD_IP:                 (v1:status.podIP)
      KB_POD_IPS:                (v1:status.podIPs)
      KB_HOSTIP:                 (v1:status.hostIP)
      KB_PODIP:                  (v1:status.podIP)
      KB_PODIPS:                 (v1:status.podIPs)
      KB_POD_FQDN:              $(KB_POD_NAME).redis-jwrrzl-redis-headless.$(KB_NAMESPACE).svc
      SERVICE_PORT:             6379
      REDIS_REPL_USER:          kbreplicator
      REDIS_REPL_PASSWORD:      $(REDIS_DEFAULT_PASSWORD)
      REDIS_SENTINEL_USER:      $(REDIS_REPL_USER)-sentinel
      REDIS_SENTINEL_PASSWORD:  $(REDIS_DEFAULT_PASSWORD)
    Mounts:
      /data from data (rw)
      /etc/conf from redis-config (rw)
      /etc/redis from redis-conf (rw)
      /kb-podinfo from pod-info (rw)
      /scripts from scripts (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-5bmdc (ro)
  metrics:
    Container ID:  containerd://f0d713c4137e919523e04f34d8b6be9de9e7b63ea5c5754672c4c70794e1078d
    Image:         apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/agamotto:0.1.2-beta.1
    Image ID:      apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/agamotto@sha256:01d06a16a2e96d0c5e1ac97db40353abad8ee73f34f9f7d148984e7de4ff0bd5
    Port:          9121/TCP
    Host Port:     0/TCP
    Command:
      /bin/agamotto
      --config=/opt/conf/metrics-config.yaml
    State:          Running
      Started:      Tue, 15 Oct 2024 17:32:58 +0800
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     0
      memory:  0
    Requests:
      cpu:     0
      memory:  0
    Environment Variables from:
      redis-jwrrzl-redis-env      ConfigMap  Optional: false
      redis-jwrrzl-redis-rsm-env  ConfigMap  Optional: false
    Environment:
      REDIS_DEFAULT_USER:      <set to the key 'username' in secret 'redis-jwrrzl-redis-account-default'>           Optional: false
      REDIS_DEFAULT_PASSWORD:  <set to the key 'password' in secret 'redis-jwrrzl-redis-account-default'>           Optional: false
      SENTINEL_USER:           <set to the key 'username' in secret 'redis-jwrrzl-redis-sentinel-account-default'>  Optional: false
      SENTINEL_PASSWORD:       <set to the key 'password' in secret 'redis-jwrrzl-redis-sentinel-account-default'>  Optional: false
      KB_POD_NAME:             redis-jwrrzl-redis-1 (v1:metadata.name)
      KB_POD_UID:               (v1:metadata.uid)
      KB_NAMESPACE:            default (v1:metadata.namespace)
      KB_SA_NAME:               (v1:spec.serviceAccountName)
      KB_NODENAME:              (v1:spec.nodeName)
      KB_HOST_IP:               (v1:status.hostIP)
      KB_POD_IP:                (v1:status.podIP)
      KB_POD_IPS:               (v1:status.podIPs)
      KB_HOSTIP:                (v1:status.hostIP)
      KB_PODIP:                 (v1:status.podIP)
      KB_PODIPS:                (v1:status.podIPs)
      KB_POD_FQDN:             $(KB_POD_NAME).redis-jwrrzl-redis-headless.$(KB_NAMESPACE).svc
      ENDPOINT:                localhost:6379
      REDIS_USER:              $(REDIS_DEFAULT_USER)
      REDIS_PASSWORD:          $(REDIS_DEFAULT_PASSWORD)
    Mounts:
      /opt/conf from redis-metrics-config (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-5bmdc (ro)
  lorry:
    Container ID:  containerd://72bb89fe2db6668a946e803e2e5655e59796db6755d92a600ae0fcea6a873969
    Image:         apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/redis-stack-server:7.2.0-v10
    Image ID:      apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/redis-stack-server@sha256:f55712f253ffafedd9f403cee24bc6644a26e058d48a65037bc64b6d31a86349
    Ports:         3501/TCP, 50001/TCP
    Host Ports:    0/TCP, 0/TCP
    Command:
      /kubeblocks/lorry
      --port
      3501
      --grpcport
      50001
      --config-path
      /kubeblocks/config/lorry/components/
    State:          Running
      Started:      Tue, 15 Oct 2024 17:33:00 +0800
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     0
      memory:  0
    Requests:
      cpu:      0
      memory:   0
    Readiness:  http-get http://:3501/v1.0/checkrole delay=0s timeout=1s period=1s #success=1 #failure=3
    Startup:    tcp-socket :3501 delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment Variables from:
      redis-jwrrzl-redis-env      ConfigMap  Optional: false
      redis-jwrrzl-redis-rsm-env  ConfigMap  Optional: false
    Environment:
      REDIS_DEFAULT_USER:            <set to the key 'username' in secret 'redis-jwrrzl-redis-account-default'>           Optional: false
      REDIS_DEFAULT_PASSWORD:        <set to the key 'password' in secret 'redis-jwrrzl-redis-account-default'>           Optional: false
      SENTINEL_USER:                 <set to the key 'username' in secret 'redis-jwrrzl-redis-sentinel-account-default'>  Optional: false
      SENTINEL_PASSWORD:             <set to the key 'password' in secret 'redis-jwrrzl-redis-sentinel-account-default'>  Optional: false
      KB_POD_NAME:                   redis-jwrrzl-redis-1 (v1:metadata.name)
      KB_POD_UID:                     (v1:metadata.uid)
      KB_NAMESPACE:                  default (v1:metadata.namespace)
      KB_SA_NAME:                     (v1:spec.serviceAccountName)
      KB_NODENAME:                    (v1:spec.nodeName)
      KB_HOST_IP:                     (v1:status.hostIP)
      KB_POD_IP:                      (v1:status.podIP)
      KB_POD_IPS:                     (v1:status.podIPs)
      KB_HOSTIP:                      (v1:status.hostIP)
      KB_PODIP:                       (v1:status.podIP)
      KB_PODIPS:                      (v1:status.podIPs)
      KB_POD_FQDN:                   $(KB_POD_NAME).redis-jwrrzl-redis-headless.$(KB_NAMESPACE).svc
      KB_RSM_ROLE_PROBE_PERIOD:      1
      KB_BUILTIN_HANDLER:            redis
      KB_SERVICE_USER:               <set to the key 'username' in secret 'redis-jwrrzl-redis-account-default'>  Optional: false
      KB_SERVICE_PASSWORD:           <set to the key 'password' in secret 'redis-jwrrzl-redis-account-default'>  Optional: false
      KB_SERVICE_PORT:               6379
      KB_DATA_PATH:                  /data
      KB_ACTION_COMMANDS:            {"accountProvision":["sh","-c","redis-cli -h $(KB_ACCOUNT_ENDPOINT) -a $REDIS_DEFAULT_PASSWORD $(KB_ACCOUNT_STATEMENT) \u0026\u0026 redis-cli -h $(KB_ACCOUNT_ENDPOINT) -a $REDIS_DEFAULT_PASSWORD acl save "],"postProvision":["/bin/bash","-c","/scripts/redis-register-to-sentinel.sh"]}
      SERVICE_PORT:                  6379
      REDIS_REPL_USER:               kbreplicator
      REDIS_REPL_PASSWORD:           $(REDIS_DEFAULT_PASSWORD)
      REDIS_SENTINEL_USER:           $(REDIS_REPL_USER)-sentinel
      REDIS_SENTINEL_PASSWORD:       $(REDIS_DEFAULT_PASSWORD)
      KB_RSM_ACTION_SVC_LIST:        null
      KB_RSM_ROLE_UPDATE_MECHANISM:  DirectAPIServerEventUpdate
      KB_RSM_ROLE_PROBE_TIMEOUT:     1
      KB_CLUSTER_NAME:                (v1:metadata.labels['app.kubernetes.io/instance'])
      KB_COMP_NAME:                   (v1:metadata.labels['apps.kubeblocks.io/component-name'])
      KB_SERVICE_CHARACTER_TYPE:     redis
    Mounts:
      /data from data (rw)
      /kubeblocks from kubeblocks (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-5bmdc (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  pod-info:
    Type:  DownwardAPI (a volume populated by information about the pod)
    Items:
      metadata.annotations['rs.apps.kubeblocks.io/primary'] -> primary-pod
      metadata.annotations['apps.kubeblocks.io/component-replicas'] -> component-replicas
  redis-metrics-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      redis-jwrrzl-redis-redis-metrics-config
    Optional:  false
  redis-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      redis-jwrrzl-redis-redis-replication-config
    Optional:  false
  scripts:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      redis-jwrrzl-redis-redis-scripts
    Optional:  false
  data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  data-redis-jwrrzl-redis-1
    ReadOnly:   false
  redis-conf:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  kubeblocks:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  kube-api-access-5bmdc:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 kb-data=true:NoSchedule
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Scheduled  13m                  default-scheduler  Successfully assigned default/redis-jwrrzl-redis-1 to 172.16.0.72
  Normal   Pulling    13m                  kubelet            Pulling image "apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/kubeblocks-tools:0.9.1"
  Normal   Pulled     13m                  kubelet            Successfully pulled image "apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/kubeblocks-tools:0.9.1" in 460ms (460ms including waiting)
  Normal   Created    13m                  kubelet            Created container init-lorry
  Normal   Started    13m                  kubelet            Started container init-lorry
  Normal   Pulled     13m                  kubelet            Successfully pulled image "apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/redis-stack-server:7.2.0-v10" in 1.973s (1.973s including waiting)
  Normal   Pulling    13m                  kubelet            Pulling image "apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/agamotto:0.1.2-beta.1"
  Normal   Pulled     13m                  kubelet            Successfully pulled image "apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/agamotto:0.1.2-beta.1" in 1.955s (1.955s including waiting)
  Normal   Started    13m                  kubelet            Started container metrics
  Normal   Created    13m                  kubelet            Created container metrics
  Normal   Pulling    13m                  kubelet            Pulling image "apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/redis-stack-server:7.2.0-v10"
  Normal   Pulled     13m                  kubelet            Successfully pulled image "apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/redis-stack-server:7.2.0-v10" in 1.64s (1.64s including waiting)
  Normal   Created    13m                  kubelet            Created container lorry
  Normal   Started    13m                  kubelet            Started container lorry
  Normal   checkRole  13m                  lorry              {"event":"Success","operation":"checkRole","originalRole":"waitForStart","role":"secondary"}
  Normal   checkRole  12m                  lorry              {"event":"Success","operation":"checkRole","originalRole":"secondary","role":"{\"term\":\"1728984792862705\",\"PodRoleNamePairs\":[{\"podName\":\"redis-jwrrzl-redis-1\",\"roleName\":\"primary\",\"podUid\":\"578c2195-0e2e-4e31-8002-1057f17eb14d\"}]}"}
  Normal   checkRole  12m                  lorry              {"operation":"checkRole","originalRole":"primary"}
  Normal   Pulled     12m                  kubelet            Successfully pulled image "apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/redis-stack-server:7.2.0-v10" in 372ms (372ms including waiting)
  Normal   Started    12m (x2 over 13m)    kubelet            Started container redis
  Normal   Pulling    11m (x3 over 13m)    kubelet            Pulling image "apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/redis-stack-server:7.2.0-v10"
  Normal   Pulled     11m                  kubelet            Successfully pulled image "apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/redis-stack-server:7.2.0-v10" in 400ms (400ms including waiting)
  Normal   Created    11m (x3 over 13m)    kubelet            Created container redis
  Normal   checkRole  7m3s                 lorry              {"operation":"checkRole","originalRole":"primary"}
  Warning  BackOff    3m6s (x47 over 12m)  kubelet            Back-off restarting failed container redis in pod redis-jwrrzl-redis-1_default(578c2195-0e2e-4e31-8002-1057f17eb14d)
  Normal   checkRole  2m3s                 lorry              {"operation":"checkRole","originalRole":"primary"}

logs pod

kubectl logs redis-jwrrzl-redis-1  redis
+ declare -g primary
+ declare -g primary_port
+ declare -g default_initialize_pod_ordinal
+ declare -g headless_postfix=headless
+ parse_redis_advertised_svc_if_exist redis-jwrrzl-redis-1
+ local pod_name=redis-jwrrzl-redis-1
+ [[ -z '' ]]
+ echo 'Environment variable REDIS_ADVERTISED_PORT not found. Ignoring.'
+ return 0
+ build_redis_conf
+ load_redis_template_conf
Environment variable REDIS_ADVERTISED_PORT not found. Ignoring.
+ echo 'include /etc/conf/redis.conf'
+ build_announce_ip_and_port
+ '[' -n '' ']'
+ kb_pod_fqdn=redis-jwrrzl-redis-1.redis-jwrrzl-redis-headless.default.svc
+ echo 'redis use kb pod fqdn redis-jwrrzl-redis-1.redis-jwrrzl-redis-headless.default.svc to announce'
+ echo 'replica-announce-ip redis-jwrrzl-redis-1.redis-jwrrzl-redis-headless.default.svc'
redis use kb pod fqdn redis-jwrrzl-redis-1.redis-jwrrzl-redis-headless.default.svc to announce
+ build_redis_service_port
+ service_port=6379
+ '[' -n 6379 ']'
+ service_port=6379
+ echo 'port 6379'
+ build_replicaof_config
+ init_or_get_primary_node
+ init_or_get_primary_from_redis_sentinel
+ '[' -z redis-jwrrzl-redis-sentinel ']'
+ '[' -z redis-jwrrzl-redis-sentinel-0,redis-jwrrzl-redis-sentinel-1,redis-jwrrzl-redis-sentinel-2 ']'
+ '[' -z redis-jwrrzl-redis-sentinel-headless ']'
+ old_ifs=' 	
'
+ IFS=,
+ set -f
+ read -ra sentinel_pod_list
+ set +f
+ IFS=' 	
'
+ declare -A master_count_map
+ local first_redis_primary_host=
+ local first_redis_primary_port=
+ for sentinel_pod in "${sentinel_pod_list[@]}"
+ sentinel_pod_fqdn=redis-jwrrzl-redis-sentinel-0.redis-jwrrzl-redis-sentinel-headless
+ retry_get_master_addr_by_name_from_sentinel redis-jwrrzl-redis-sentinel-0.redis-jwrrzl-redis-sentinel-headless
+ local sentinel_pod_fqdn=redis-jwrrzl-redis-sentinel-0.redis-jwrrzl-redis-sentinel-headless
+ local retry_count=0
+ local max_retry=3
+ local retry_delay=2
+ local timeout_value=5
+ '[' 0 -lt 3 ']'
+ set +x
execute command: timeout 5 redis-cli -h redis-jwrrzl-redis-sentinel-0.redis-jwrrzl-redis-sentinel-headless -p 26379 -a ******** sentinel get-master-addr-by-name redis-jwrrzl-redis
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
+ exit_code=0
+ '[' 0 -eq 0 ']'
+ old_ifs=' 	
'
+ IFS='
'
+ set -f
+ read -r -d '' -a REDIS_SENTINEL_PRIMARY_INFO
+ set +f
+ IFS=' 	
'
+ '[' 2 -eq 2 ']'
+ '[' -n redis-jwrrzl-redis-0.redis-jwrrzl-redis-headless.default.svc ']'
+ '[' -n 6379 ']'
+ echo 'Successfully retrieved primary info from sentinel: redis-jwrrzl-redis-sentinel-0.redis-jwrrzl-redis-sentinel-headless'
+ return 0
Successfully retrieved primary info from sentinel: redis-jwrrzl-redis-sentinel-0.redis-jwrrzl-redis-sentinel-headless
+ echo 'sentinel:redis-jwrrzl-redis-sentinel-0.redis-jwrrzl-redis-sentinel-headless has master info: redis-jwrrzl-redis-0.redis-jwrrzl-redis-headless.default.svc 6379'
+ '[' 2 -ne 2 ']'
sentinel:redis-jwrrzl-redis-sentinel-0.redis-jwrrzl-redis-sentinel-headless has master info: redis-jwrrzl-redis-0.redis-jwrrzl-redis-headless.default.svc 6379
+ '[' -z redis-jwrrzl-redis-0.redis-jwrrzl-redis-headless.default.svc ']'
+ '[' -z 6379 ']'
+ host_port_key=redis-jwrrzl-redis-0.redis-jwrrzl-redis-headless.default.svc:6379
+ master_count_map[$host_port_key]=1
+ [[ -z '' ]]
+ [[ -z '' ]]
+ first_redis_primary_host=redis-jwrrzl-redis-0.redis-jwrrzl-redis-headless.default.svc
+ first_redis_primary_port=6379
+ '[' redis-jwrrzl-redis-0.redis-jwrrzl-redis-headless.default.svc '!=' redis-jwrrzl-redis-0.redis-jwrrzl-redis-headless.default.svc ']'
+ '[' 6379 '!=' 6379 ']'
+ for sentinel_pod in "${sentinel_pod_list[@]}"
+ sentinel_pod_fqdn=redis-jwrrzl-redis-sentinel-1.redis-jwrrzl-redis-sentinel-headless
+ retry_get_master_addr_by_name_from_sentinel redis-jwrrzl-redis-sentinel-1.redis-jwrrzl-redis-sentinel-headless
+ local sentinel_pod_fqdn=redis-jwrrzl-redis-sentinel-1.redis-jwrrzl-redis-sentinel-headless
+ local retry_count=0
+ local max_retry=3
+ local retry_delay=2
+ local timeout_value=5
+ '[' 0 -lt 3 ']'
+ set +x
execute command: timeout 5 redis-cli -h redis-jwrrzl-redis-sentinel-1.redis-jwrrzl-redis-sentinel-headless -p 26379 -a ******** sentinel get-master-addr-by-name redis-jwrrzl-redis
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
+ exit_code=0
+ '[' 0 -eq 0 ']'
+ old_ifs=' 	
'
+ IFS='
'
+ set -f
+ read -r -d '' -a REDIS_SENTINEL_PRIMARY_INFO
+ set +f
+ IFS=' 	
'
+ '[' 2 -eq 2 ']'
+ '[' -n redis-jwrrzl-redis-0.redis-jwrrzl-redis-headless.default.svc ']'
+ '[' -n 6379 ']'
Successfully retrieved primary info from sentinel: redis-jwrrzl-redis-sentinel-1.redis-jwrrzl-redis-sentinel-headless
+ echo 'Successfully retrieved primary info from sentinel: redis-jwrrzl-redis-sentinel-1.redis-jwrrzl-redis-sentinel-headless'
+ return 0
sentinel:redis-jwrrzl-redis-sentinel-1.redis-jwrrzl-redis-sentinel-headless has master info: redis-jwrrzl-redis-0.redis-jwrrzl-redis-headless.default.svc 6379
+ echo 'sentinel:redis-jwrrzl-redis-sentinel-1.redis-jwrrzl-redis-sentinel-headless has master info: redis-jwrrzl-redis-0.redis-jwrrzl-redis-headless.default.svc 6379'
+ '[' 2 -ne 2 ']'
+ '[' -z redis-jwrrzl-redis-0.redis-jwrrzl-redis-headless.default.svc ']'
+ '[' -z 6379 ']'
+ host_port_key=redis-jwrrzl-redis-0.redis-jwrrzl-redis-headless.default.svc:6379
+ master_count_map[$host_port_key]=2
+ [[ -z redis-jwrrzl-redis-0.redis-jwrrzl-redis-headless.default.svc ]]
+ '[' redis-jwrrzl-redis-0.redis-jwrrzl-redis-headless.default.svc '!=' redis-jwrrzl-redis-0.redis-jwrrzl-redis-headless.default.svc ']'
+ '[' 6379 '!=' 6379 ']'
+ for sentinel_pod in "${sentinel_pod_list[@]}"
+ sentinel_pod_fqdn=redis-jwrrzl-redis-sentinel-2.redis-jwrrzl-redis-sentinel-headless
+ retry_get_master_addr_by_name_from_sentinel redis-jwrrzl-redis-sentinel-2.redis-jwrrzl-redis-sentinel-headless
+ local sentinel_pod_fqdn=redis-jwrrzl-redis-sentinel-2.redis-jwrrzl-redis-sentinel-headless
+ local retry_count=0
+ local max_retry=3
+ local retry_delay=2
+ local timeout_value=5
+ '[' 0 -lt 3 ']'
+ set +x
execute command: timeout 5 redis-cli -h redis-jwrrzl-redis-sentinel-2.redis-jwrrzl-redis-sentinel-headless -p 26379 -a ******** sentinel get-master-addr-by-name redis-jwrrzl-redis
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
+ exit_code=0
+ '[' 0 -eq 0 ']'
+ old_ifs=' 	
'
+ IFS='
'
+ set -f
+ read -r -d '' -a REDIS_SENTINEL_PRIMARY_INFO
+ set +f
+ IFS=' 	
'
+ '[' 2 -eq 2 ']'
+ '[' -n redis-jwrrzl-redis-0.redis-jwrrzl-redis-headless.default.svc ']'
+ '[' -n 6379 ']'
+ echo 'Successfully retrieved primary info from sentinel: redis-jwrrzl-redis-sentinel-2.redis-jwrrzl-redis-sentinel-headless'
+ return 0
Successfully retrieved primary info from sentinel: redis-jwrrzl-redis-sentinel-2.redis-jwrrzl-redis-sentinel-headless
sentinel:redis-jwrrzl-redis-sentinel-2.redis-jwrrzl-redis-sentinel-headless has master info: redis-jwrrzl-redis-0.redis-jwrrzl-redis-headless.default.svc 6379
+ echo 'sentinel:redis-jwrrzl-redis-sentinel-2.redis-jwrrzl-redis-sentinel-headless has master info: redis-jwrrzl-redis-0.redis-jwrrzl-redis-headless.default.svc 6379'
+ '[' 2 -ne 2 ']'
+ '[' -z redis-jwrrzl-redis-0.redis-jwrrzl-redis-headless.default.svc ']'
+ '[' -z 6379 ']'
+ host_port_key=redis-jwrrzl-redis-0.redis-jwrrzl-redis-headless.default.svc:6379
+ master_count_map[$host_port_key]=3
+ [[ -z redis-jwrrzl-redis-0.redis-jwrrzl-redis-headless.default.svc ]]
+ '[' redis-jwrrzl-redis-0.redis-jwrrzl-redis-headless.default.svc '!=' redis-jwrrzl-redis-0.redis-jwrrzl-redis-headless.default.svc ']'
+ '[' 6379 '!=' 6379 ']'
+ echo 'get all primary info from redis sentinel master_count_map: 3'
+ '[' 1 -eq 0 ']'
+ max_count=0
get all primary info from redis sentinel master_count_map: 3
+ for host_port in "${!master_count_map[@]}"
+ ((  3 > max_count  ))
+ max_count=3
++ echo redis-jwrrzl-redis-0.redis-jwrrzl-redis-headless.default.svc:6379
++ cut -d: -f1
+ primary=redis-jwrrzl-redis-0.redis-jwrrzl-redis-headless.default.svc
++ echo redis-jwrrzl-redis-0.redis-jwrrzl-redis-headless.default.svc:6379
++ cut -d: -f2
+ primary_port=6379
+ check_current_pod_is_primary
+ current_pod=redis-jwrrzl-redis-1.redis-jwrrzl-redis
+ [[ redis-jwrrzl-redis-0.redis-jwrrzl-redis-headless.default.svc == *\r\e\d\i\s\-\j\w\r\r\z\l\-\r\e\d\i\s\-\1\.\r\e\d\i\s\-\j\w\r\r\z\l\-\r\e\d\i\s* ]]
+ '[' -n '' ']'
+ [[ redis-jwrrzl-redis-0.redis-jwrrzl-redis-headless.default.svc == \1\7\2\.\1\6\.\0\.\7\4 ]]
+ return 1
+ echo 'replicaof redis-jwrrzl-redis-0.redis-jwrrzl-redis-headless.default.svc 6379'
+ rebuild_redis_acl_file
+ '[' -f /data/users.acl ']'
+ sed -i '/user default on/d' /data/users.acl
+ sed -i '/user kbreplicator on/d' /data/users.acl
+ sed -i '/user kbreplicator-sentinel on/d' /data/users.acl
+ build_redis_default_accounts
+ set +x
+ echo 'aclfile /data/users.acl'
+ echo 'build default accounts succeeded!'
build default accounts succeeded!
+ start_redis_server
+ exec_cmd='exec redis-server /etc/redis/redis.conf'
+ '[' -f /opt/redis-stack/lib/redisearch.so ']'
+ exec_cmd='exec redis-server /etc/redis/redis.conf --loadmodule /opt/redis-stack/lib/redisearch.so '
+ '[' -f /opt/redis-stack/lib/redistimeseries.so ']'
+ exec_cmd='exec redis-server /etc/redis/redis.conf --loadmodule /opt/redis-stack/lib/redisearch.so  --loadmodule /opt/redis-stack/lib/redistimeseries.so '
+ '[' -f /opt/redis-stack/lib/rejson.so ']'
+ exec_cmd='exec redis-server /etc/redis/redis.conf --loadmodule /opt/redis-stack/lib/redisearch.so  --loadmodule /opt/redis-stack/lib/redistimeseries.so  --loadmodule /opt/redis-stack/lib/rejson.so '
+ '[' -f /opt/redis-stack/lib/redisbloom.so ']'
+ exec_cmd='exec redis-server /etc/redis/redis.conf --loadmodule /opt/redis-stack/lib/redisearch.so  --loadmodule /opt/redis-stack/lib/redistimeseries.so  --loadmodule /opt/redis-stack/lib/rejson.so  --loadmodule /opt/redis-stack/lib/redisbloom.so '
+ '[' -f /opt/redis-stack/lib/redisgraph.so ']'
+ '[' -f /opt/redis-stack/lib/rediscompat.so ']'
+ exec_cmd='exec redis-server /etc/redis/redis.conf --loadmodule /opt/redis-stack/lib/redisearch.so  --loadmodule /opt/redis-stack/lib/redistimeseries.so  --loadmodule /opt/redis-stack/lib/rejson.so  --loadmodule /opt/redis-stack/lib/redisbloom.so  --loadmodule /opt/redis-stack/lib/rediscompat.so'
+ '[' -f /opt/redis-stack/lib/redisgears.so ']'
+ exec_cmd='exec redis-server /etc/redis/redis.conf --loadmodule /opt/redis-stack/lib/redisearch.so  --loadmodule /opt/redis-stack/lib/redistimeseries.so  --loadmodule /opt/redis-stack/lib/rejson.so  --loadmodule /opt/redis-stack/lib/redisbloom.so  --loadmodule /opt/redis-stack/lib/rediscompat.so --loadmodule /opt/redis-stack/lib/redisgears.so v8-plugin-path /opt/redis-stack/lib/libredisgears_v8_plugin.so '
+ echo 'Starting redis server cmd: exec redis-server /etc/redis/redis.conf --loadmodule /opt/redis-stack/lib/redisearch.so  --loadmodule /opt/redis-stack/lib/redistimeseries.so  --loadmodule /opt/redis-stack/lib/rejson.so  --loadmodule /opt/redis-stack/lib/redisbloom.so  --loadmodule /opt/redis-stack/lib/rediscompat.so --loadmodule /opt/redis-stack/lib/redisgears.so v8-plugin-path /opt/redis-stack/lib/libredisgears_v8_plugin.so '
Starting redis server cmd: exec redis-server /etc/redis/redis.conf --loadmodule /opt/redis-stack/lib/redisearch.so  --loadmodule /opt/redis-stack/lib/redistimeseries.so  --loadmodule /opt/redis-stack/lib/rejson.so  --loadmodule /opt/redis-stack/lib/redisbloom.so  --loadmodule /opt/redis-stack/lib/rediscompat.so --loadmodule /opt/redis-stack/lib/redisgears.so v8-plugin-path /opt/redis-stack/lib/libredisgears_v8_plugin.so 
+ eval 'exec redis-server /etc/redis/redis.conf --loadmodule /opt/redis-stack/lib/redisearch.so  --loadmodule /opt/redis-stack/lib/redistimeseries.so  --loadmodule /opt/redis-stack/lib/rejson.so  --loadmodule /opt/redis-stack/lib/redisbloom.so  --loadmodule /opt/redis-stack/lib/rediscompat.so --loadmodule /opt/redis-stack/lib/redisgears.so v8-plugin-path /opt/redis-stack/lib/libredisgears_v8_plugin.so '
++ exec redis-server /etc/redis/redis.conf --loadmodule /opt/redis-stack/lib/redisearch.so --loadmodule /opt/redis-stack/lib/redistimeseries.so --loadmodule /opt/redis-stack/lib/rejson.so --loadmodule /opt/redis-stack/lib/redisbloom.so --loadmodule /opt/redis-stack/lib/rediscompat.so --loadmodule /opt/redis-stack/lib/redisgears.so v8-plugin-path /opt/redis-stack/lib/libredisgears_v8_plugin.so
➜  ~

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

  • OS: [e.g. iOS]
  • Browser [e.g. chrome, safari]
  • Version [e.g. 22]

Additional context
Add any other context about the problem here.
@JashBook JashBook added the kind/bug Something isn't working label Oct 15, 2024
@JashBook JashBook added this to the Release 0.9.2 milestone Oct 15, 2024
@JashBook JashBook changed the title [BUG] redis pod CrashLoopBackOff [BUG] redis pod CrashLoopBackOff after kill 1 Oct 15, 2024
@github-actions GitHub Actions
Copy link

This issue has been marked as stale because it has been open for 30 days with no activity

@github-actions github-actions bot added the Stale label Nov 18, 2024
@github-actions github-actions bot modified the milestones: Release 0.9.2, Release 0.9.3 Jan 14, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Y-Rookie Y-Rookie

Labels
kind/bug Something isn't working Stale
Projects
None yet
Milestone
Release 0.9.3
Development

No branches or pull requests
2 participants
@Y-Rookie @JashBook