Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG]redis cluster with nodeport cluster nodes info not correct when hscale replica #7100

Closed
ahjing99 opened this issue Apr 19, 2024 · 2 comments
Assignees
Labels
kind/bug Something isn't working Stale
Milestone

Comments

@ahjing99
Copy link
Collaborator

kbcli version
Kubernetes: v1.28.7-gke.1026000
KubeBlocks: 0.9.0-beta.10
kbcli: 0.9.0-beta.1

  1. Create cluster , kubectl apply -f cluster-shard.yaml
apiVersion: apps.kubeblocks.io/v1alpha1
kind: Cluster
metadata:
  name: redis-cluster
  namespace: default
spec:
  # Specifies the cluster termination policy.
  # - DoNotTerminate will block delete operation.
  # - Halt will delete workload resources such as statefulset, deployment workloads but keep PVCs.
  # - Delete is based on Halt and deletes PVCs.
  # - WipeOut is based on Delete and wipe out all volume snapshots and snapshot data from backup storage location.
  terminationPolicy: Delete
  # List of ShardingSpecs used to define the components that make up a cluster.
  # ComponentSpecs and ShardingSpecs cannot both be empty at the same time.
  shardingSpecs:
      # Specifies the identifier for the sharding configuration.
      # This identifier is included as part of the Service DNS name and must comply with IANA Service Naming rules.
    - name: shard
      # Specifies the number of components, all of which will have identical specifications and definitions.
      # The number of replicas for each component should be defined by template.replicas.
      # The logical relationship between these components should be maintained by the components themselves
      # The number of shards should be no less than 3
      shards: 3
      # The blueprint for the components. Generates a set of components (also referred to as shards) based on this template.
      template:
        # Specifies the name of the cluster's component.
        name: redis
        # References the name of the ComponentDefinition.
        componentDef: redis-cluster
        # The cluster-level configuration is used as the default configuration of all components;
        # if the affinity and tolerations exists in a component, the component-level configuration
        # will take effect and cover the default cluster-level configuration
        affinity:
          # Specifies the anti-affinity level of pods within a component.
          # - Preferred
          # - Required
          podAntiAffinity: Preferred
          # Represents the key of node labels.
          topologyKeys:
            - kubernetes.io/hostname
          # Defines how pods are distributed across nodes.
          tenancy: SharedNode
        # Attached to tolerate any taint that matches the triple `key,value,effect` using the matching operator `operator`.
        tolerations:
          - key: kb-data
            operator: Equal
            value: "true"
            effect: NoSchedule
        # enable monitor of the component, default value is false if not claimed
        monitor: false
        # Indicates which log file takes effect
        enabledLogs:
          - running
        # if not specified, will use the default service account created along with cluster
        serviceAccountName: kb-redis-cluster
        # Specifies the number of component replicas.
        replicas: 1
        # Defines the strategy for switchover and failover when workloadType is Replication.
        switchPolicy:
          # Type specifies the type of switch policy to be applied.
          type: Noop
        # Specifies the resources requests and limits of the workload.
        resources:
          limits:
            cpu: "1"
            memory: "1.1Gi"
          requests:
            cpu: "1"
            memory: "1.1Gi"
        # Specifies to use the NodePort, need to delete the serviceVersion and services specs if you want to use container network
        serviceVersion: 7.0.6
        services:
          - name: redis-advertised
            podService: true
            #serverType
            #  - NodePort
            #  - LoadBalancer
            serviceType: NodePort
        # Provides information for statefulset.spec.volumeClaimTemplates.
        volumeClaimTemplates:
            # Refers to `clusterDefinition.spec.componentDefs.containers.volumeMounts.name`.
          - name: data
            spec:
              # Contains the desired access modes the volume should have.
              accessModes:
                - ReadWriteOnce
              # Represents the minimum resources the volume should have.
              resources:
                requests:
                  storage: 20Gi
  1. Edit the yaml ,change replicas from 1 to 2, then kubectl apply -f cluster-shard.yaml
k get cluster
NAME            CLUSTER-DEFINITION   VERSION   TERMINATION-POLICY   STATUS    AGE
redis-cluster                                  Delete               Running   21m

k get pod
NAME                        READY   STATUS    RESTARTS        AGE
redis-cluster-shard-49r-0   3/3     Running   2 (9m33s ago)   12m
redis-cluster-shard-49r-1   3/3     Running   0               11m
redis-cluster-shard-sjh-0   3/3     Running   0               12m
redis-cluster-shard-sjh-1   3/3     Running   0               11m
redis-cluster-shard-wpv-0   3/3     Running   0               12m
redis-cluster-shard-wpv-1   3/3     Running   0               11m

2 new created pod cluster nodes info is not correct

# redis-cli -h 10.128.0.30 -p 31022 -a O3605v7HsS
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
10.128.0.30:31022> cluster nodes
f5cce503559018a68ffb8f29e2a1ad83f74e9690 10.128.0.30:31843@31771,redis-cluster-shard-sjh-1.redis-cluster-shard-sjh-headless.default.svc master - 0 1713521093000 0 connected
88780f6583930a9b4a6f74f1d07d1187af3e56a9 10.128.0.30:30924@31891,redis-cluster-shard-49r-0.redis-cluster-shard-49r-headless.default.svc master - 0 1713521094486 7 connected 6827-10922 15019-16383
9315a33085454e972d2dd489fffa31900d6a9f47 10.128.0.24:30628@31979,redis-cluster-shard-49r-1.redis-cluster-shard-49r-headless.default.svc master - 0 1713521093000 7 connected
8126a4ea2434ab9326e221a38b3e16e14d0dad86 10.128.0.30:31022@31002,redis-cluster-shard-wpv-1.redis-cluster-shard-wpv-headless.default.svc myself,slave 42357755c4a03bb37f759348dfdd132738b6a478 0 1713521091000 6 connected
bf35cb9cdaa340a18f19fa2fc420370b1827ad07 10.128.0.28:30301@32062,redis-cluster-shard-sjh-0.redis-cluster-shard-sjh-headless.default.svc master - 0 1713521092576 5 connected 1365-5460 12288-13653
42357755c4a03bb37f759348dfdd132738b6a478 10.128.0.28:32745@32317,redis-cluster-shard-wpv-0.redis-cluster-shard-wpv-headless.default.svc master - 0 1713521093481 6 connected 0-1364 5461-6826 10923-12287 13654-15018

# redis-cli -h 10.128.0.24 -p 30628 -a O3605v7HsS
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
10.128.0.24:30628> cluster nodes
9315a33085454e972d2dd489fffa31900d6a9f47 10.128.0.24:30628@31979,redis-cluster-shard-49r-1.redis-cluster-shard-49r-headless.default.svc myself,master - 0 1713520872839 0 connected
10.128.0.24:30628> exit
# redis-cli -h 10.128.0.30 -p 31843 -a O3605v7HsS
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
10.128.0.30:31843>  cluster nodes
f5cce503559018a68ffb8f29e2a1ad83f74e9690 10.128.0.24:31843@31771,redis-cluster-shard-sjh-1.redis-cluster-shard-sjh-headless.default.svc myself,master - 0 1713520894969 0 connected
  1. Logs
 k logs redis-cluster-shard-sjh-1
Defaulted container "redis-cluster" out of: redis-cluster, metrics, lorry, init-lorry (init)
+ parse_redis_cluster_advertised_svc_if_exist redis-cluster-shard-sjh-1
+ local pod_name=redis-cluster-shard-sjh-1
+ [[ -z redis-cluster-shard-sjh-redis-advertised-0:30301,redis-cluster-shard-sjh-redis-advertised-1:31843 ]]
+ [[ -z redis-cluster-shard-sjh-redis-advertised-0:32062,redis-cluster-shard-sjh-redis-advertised-1:31771 ]]
+ local port
++ parse_advertised_port redis-cluster-shard-sjh-1 redis-cluster-shard-sjh-redis-advertised-0:30301,redis-cluster-shard-sjh-redis-advertised-1:31843
++ local pod_name=redis-cluster-shard-sjh-1
++ local advertised_ports=redis-cluster-shard-sjh-redis-advertised-0:30301,redis-cluster-shard-sjh-redis-advertised-1:31843
++ local pod_name_ordinal
++ local found=false
+++ extract_ordinal_from_object_name redis-cluster-shard-sjh-1
+++ local object_name=redis-cluster-shard-sjh-1
+++ local ordinal=1
+++ echo 1
++ pod_name_ordinal=1
++ IFS=,
++ read -ra ports_array
++ for entry in "${ports_array[@]}"
++ IFS=:
++ read -ra parts
++ local svc_name=redis-cluster-shard-sjh-redis-advertised-0
++ local port=30301
++ local svc_name_ordinal
+++ extract_ordinal_from_object_name redis-cluster-shard-sjh-redis-advertised-0
+++ local object_name=redis-cluster-shard-sjh-redis-advertised-0
+++ local ordinal=0
+++ echo 0
++ svc_name_ordinal=0
++ [[ 0 == \1 ]]
++ for entry in "${ports_array[@]}"
++ IFS=:
++ read -ra parts
++ local svc_name=redis-cluster-shard-sjh-redis-advertised-1
++ local port=31843
++ local svc_name_ordinal
+++ extract_ordinal_from_object_name redis-cluster-shard-sjh-redis-advertised-1
+++ local object_name=redis-cluster-shard-sjh-redis-advertised-1
+++ local ordinal=1
+++ echo 1
++ svc_name_ordinal=1
++ [[ 1 == \1 ]]
++ echo 31843
++ found=true
++ return 0
+ port=31843
+ [[ 0 -ne 0 ]]
+ [[ -z 31843 ]]
+ redis_advertised_svc_port_value=31843
+ redis_advertised_svc_host_value=10.128.0.24
+ [[ -n redis-cluster-shard-sjh-redis-advertised-0:32062,redis-cluster-shard-sjh-redis-advertised-1:31771 ]]
++ parse_advertised_port redis-cluster-shard-sjh-1 redis-cluster-shard-sjh-redis-advertised-0:32062,redis-cluster-shard-sjh-redis-advertised-1:31771
++ local pod_name=redis-cluster-shard-sjh-1
++ local advertised_ports=redis-cluster-shard-sjh-redis-advertised-0:32062,redis-cluster-shard-sjh-redis-advertised-1:31771
++ local pod_name_ordinal
++ local found=false
+++ extract_ordinal_from_object_name redis-cluster-shard-sjh-1
+++ local object_name=redis-cluster-shard-sjh-1
+++ local ordinal=1
+++ echo 1
++ pod_name_ordinal=1
++ IFS=,
++ read -ra ports_array
++ for entry in "${ports_array[@]}"
++ IFS=:
++ read -ra parts
++ local svc_name=redis-cluster-shard-sjh-redis-advertised-0
++ local port=32062
++ local svc_name_ordinal
+++ extract_ordinal_from_object_name redis-cluster-shard-sjh-redis-advertised-0
+++ local object_name=redis-cluster-shard-sjh-redis-advertised-0
+++ local ordinal=0
+++ echo 0
++ svc_name_ordinal=0
++ [[ 0 == \1 ]]
++ for entry in "${ports_array[@]}"
++ IFS=:
++ read -ra parts
++ local svc_name=redis-cluster-shard-sjh-redis-advertised-1
++ local port=31771
++ local svc_name_ordinal
+++ extract_ordinal_from_object_name redis-cluster-shard-sjh-redis-advertised-1
+++ local object_name=redis-cluster-shard-sjh-redis-advertised-1
+++ local ordinal=1
+++ echo 1
++ svc_name_ordinal=1
++ [[ 1 == \1 ]]
++ echo 31771
++ found=true
++ return 0
+ port=31771
+ [[ 0 -ne 0 ]]
+ [[ -z 31771 ]]
+ redis_advertised_svc_bus_port_value=31771
+ build_redis_conf
+ load_redis_template_conf
+ echo 'include /etc/conf/redis.conf'
+ build_redis_cluster_service_port
+ service_port=6379
+ cluster_bus_port=16379
+ '[' -n 6379 ']'
+ service_port=6379
+ '[' -n 16379 ']'
+ cluster_bus_port=16379
+ echo 'port 6379'
+ echo 'cluster-port 16379'
+ build_announce_ip_and_port
+ '[' -n 10.128.0.24 ']'
+ '[' -n 31843 ']'
+ echo 'redis use advertised svc 10.128.0.24:31843 to announce'
+ echo 'replica-announce-port 31843'
+ echo 'replica-announce-ip 10.128.0.24'
+ build_cluster_announce_info
redis use advertised svc 10.128.0.24:31843 to announce
redis cluster use advertised svc 10.128.0.24:31843@31771 to announce
+ kb_pod_fqdn=redis-cluster-shard-sjh-1.redis-cluster-shard-sjh-headless.default.svc
+ '[' -n 10.128.0.24 ']'
+ '[' -n 31843 ']'
+ '[' -n 31771 ']'
+ echo 'redis cluster use advertised svc 10.128.0.24:31843@31771 to announce'
+ echo 'cluster-announce-ip 10.128.0.24'
+ echo 'cluster-announce-port 31843'
+ echo 'cluster-announce-bus-port 31771'
+ echo 'cluster-announce-hostname redis-cluster-shard-sjh-1.redis-cluster-shard-sjh-headless.default.svc'
+ echo 'cluster-preferred-endpoint-type ip'
+ rebuild_redis_acl_file
+ '[' -f /data/users.acl ']'
+ sed -i '/user default on/d' /data/users.acl
+ sed -i '/user kbreplicator on/d' /data/users.acl
+ sed -i '/user  on/d' /data/users.acl
+ build_redis_default_accounts
+ '[' -n O3605v7HsS ']'
+ echo 'masteruser kbreplicator'
+ echo 'masterauth O3605v7HsS'
+ echo 'user kbreplicator on +psync +replconf +ping >O3605v7HsS'
+ '[' '!' -z O3605v7HsS ']'
+ echo 'protected-mode yes'
+ echo 'user default on >O3605v7HsS ~* &* +@all '
+ echo 'aclfile /data/users.acl'
+ start_redis_server
+ exec redis-server /etc/redis/redis.conf --loadmodule /opt/redis-stack/lib/redisearch.so --loadmodule /opt/redis-stack/lib/redisgraph.so --loadmodule /opt/redis-stack/lib/redistimeseries.so --loadmodule /opt/redis-stack/lib/rejson.so --loadmodule /opt/redis-stack/lib/redisbloom.so
+ scale_redis_cluster_replica
+ '[' -n O3605v7HsS ']'
+ retry redis-cli -h 127.0.0.1 -p 6379 -a O3605v7HsS ping
+ local max_attempts=20
+ local attempt=1
+ redis-cli -h 127.0.0.1 -p 6379 -a O3605v7HsS ping
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
Could not connect to Redis at 127.0.0.1:6379: Connection refused
+ '[' 1 -eq 20 ']'
+ echo 'Command '\''redis-cli -h 127.0.0.1 -p 6379 -a O3605v7HsS ping'\'' failed. Attempt 1 of 20. Retrying in 5 seconds...'
+ attempt=2
+ sleep 3
Command 'redis-cli -h 127.0.0.1 -p 6379 -a O3605v7HsS ping' failed. Attempt 1 of 20. Retrying in 5 seconds...
+ redis-cli -h 127.0.0.1 -p 6379 -a O3605v7HsS ping
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
PONG
+ '[' 2 -eq 20 ']'
+ current_pod_name=redis-cluster-shard-sjh-1
+ current_pod_fqdn=redis-cluster-shard-sjh-1.redis-cluster-shard-sjh-headless.default.svc
+ '[' -n redis-cluster-shard-sjh-0 ']'
+ target_node_fqdn=redis-cluster-shard-sjh-0.redis-cluster-shard-sjh-headless.default.svc
+ get_current_comp_nodes_for_scale_out_replica redis-cluster-shard-sjh-0.redis-cluster-shard-sjh-headless.default.svc 6379
+ local cluster_node=redis-cluster-shard-sjh-0.redis-cluster-shard-sjh-headless.default.svc
+ local cluster_node_port=6379
+ '[' -z O3605v7HsS ']'
++ redis-cli -h redis-cluster-shard-sjh-0.redis-cluster-shard-sjh-headless.default.svc -p 6379 -a O3605v7HsS cluster nodes
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
+ cluster_nodes_info='42357755c4a03bb37f759348dfdd132738b6a478 10.128.0.28:32745@32317,redis-cluster-shard-wpv-0.redis-cluster-shard-wpv-headless.default.svc master - 0 1713520896949 6 connected 0-1364 5461-6826 10923-12287 13654-15018
bf35cb9cdaa340a18f19fa2fc420370b1827ad07 10.128.0.28:30301@32062,redis-cluster-shard-sjh-0.redis-cluster-shard-sjh-headless.default.svc myself,master - 0 1713520897000 5 connected 1365-5460 12288-13653
f5cce503559018a68ffb8f29e2a1ad83f74e9690 10.128.0.30:31843@31771,redis-cluster-shard-sjh-1.redis-cluster-shard-sjh-headless.default.svc slave,fail bf35cb9cdaa340a18f19fa2fc420370b1827ad07 1713520867793 1713520865783 5 connected
9315a33085454e972d2dd489fffa31900d6a9f47 10.128.0.24:30628@31979,redis-cluster-shard-49r-1.redis-cluster-shard-49r-headless.default.svc master - 0 1713520897000 7 connected
88780f6583930a9b4a6f74f1d07d1187af3e56a9 10.128.0.30:30924@31891,redis-cluster-shard-49r-0.redis-cluster-shard-49r-headless.default.svc master - 0 1713520897955 7 connected 6827-10922 15019-16383
8126a4ea2434ab9326e221a38b3e16e14d0dad86 10.128.0.30:31022@31002,redis-cluster-shard-wpv-1.redis-cluster-shard-wpv-headless.default.svc slave 42357755c4a03bb37f759348dfdd132738b6a478 1713520893935 1713520891523 6 connected'
+ current_comp_primary_node=()
+ current_comp_other_nodes=()
++ echo '42357755c4a03bb37f759348dfdd132738b6a478 10.128.0.28:32745@32317,redis-cluster-shard-wpv-0.redis-cluster-shard-wpv-headless.default.svc master - 0 1713520896949 6 connected 0-1364 5461-6826 10923-12287 13654-15018
bf35cb9cdaa340a18f19fa2fc420370b1827ad07 10.128.0.28:30301@32062,redis-cluster-shard-sjh-0.redis-cluster-shard-sjh-headless.default.svc myself,master - 0 1713520897000 5 connected 1365-5460 12288-13653
f5cce503559018a68ffb8f29e2a1ad83f74e9690 10.128.0.30:31843@31771,redis-cluster-shard-sjh-1.redis-cluster-shard-sjh-headless.default.svc slave,fail bf35cb9cdaa340a18f19fa2fc420370b1827ad07 1713520867793 1713520865783 5 connected
9315a33085454e972d2dd489fffa31900d6a9f47 10.128.0.24:30628@31979,redis-cluster-shard-49r-1.redis-cluster-shard-49r-headless.default.svc master - 0 1713520897000 7 connected
88780f6583930a9b4a6f74f1d07d1187af3e56a9 10.128.0.30:30924@31891,redis-cluster-shard-49r-0.redis-cluster-shard-49r-headless.default.svc master - 0 1713520897955 7 connected 6827-10922 15019-16383
8126a4ea2434ab9326e221a38b3e16e14d0dad86 10.128.0.30:31022@31002,redis-cluster-shard-wpv-1.redis-cluster-shard-wpv-headless.default.svc slave 42357755c4a03bb37f759348dfdd132738b6a478 1713520893935 1713520891523 6 connected'
++ wc -l
+ '[' 6 -eq 1 ']'
+ declare -A advertised_ports
+ local using_advertised_ports=false
+ '[' -n redis-cluster-shard-sjh-redis-advertised-0:30301,redis-cluster-shard-sjh-redis-advertised-1:31843 ']'
+ using_advertised_ports=true
+ IFS=,
+ read -ra ADDR
+ for i in "${ADDR[@]}"
++ echo redis-cluster-shard-sjh-redis-advertised-0:30301
++ cut -d: -f2
+ port=30301
+ advertised_ports[$port]=1
+ for i in "${ADDR[@]}"
++ echo redis-cluster-shard-sjh-redis-advertised-1:31843
++ cut -d: -f2
+ port=31843
+ advertised_ports[$port]=1
+ read -r line
++ echo '42357755c4a03bb37f759348dfdd132738b6a478 10.128.0.28:32745@32317,redis-cluster-shard-wpv-0.redis-cluster-shard-wpv-headless.default.svc master - 0 1713520896949 6 connected 0-1364 5461-6826 10923-12287 13654-15018'
++ awk '{print $2}'
+ node_ip_port_fields=10.128.0.28:32745@32317,redis-cluster-shard-wpv-0.redis-cluster-shard-wpv-headless.default.svc
++ echo 10.128.0.28:32745@32317,redis-cluster-shard-wpv-0.redis-cluster-shard-wpv-headless.default.svc
++ awk -F @ '{print $1}'
+ node_ip_port=10.128.0.28:32745
++ echo 10.128.0.28:32745@32317,redis-cluster-shard-wpv-0.redis-cluster-shard-wpv-headless.default.svc
++ awk -F @ '{print $2}'
++ awk -F , '{print $1}'
+ node_bus_port=32317
++ echo 10.128.0.28:32745@32317,redis-cluster-shard-wpv-0.redis-cluster-shard-wpv-headless.default.svc
++ awk -F @ '{print $1}'
++ cut -d: -f2
+ node_port=32745
++ echo '42357755c4a03bb37f759348dfdd132738b6a478 10.128.0.28:32745@32317,redis-cluster-shard-wpv-0.redis-cluster-shard-wpv-headless.default.svc master - 0 1713520896949 6 connected 0-1364 5461-6826 10923-12287 13654-15018'
++ awk '{print $2}'
++ awk -F , '{print $2}'
+ node_fqdn=redis-cluster-shard-wpv-0.redis-cluster-shard-wpv-headless.default.svc
++ echo '42357755c4a03bb37f759348dfdd132738b6a478 10.128.0.28:32745@32317,redis-cluster-shard-wpv-0.redis-cluster-shard-wpv-headless.default.svc master - 0 1713520896949 6 connected 0-1364 5461-6826 10923-12287 13654-15018'
++ awk '{print $3}'
+ node_role=master
+ true
+ [[ -n '' ]]
+ read -r line
++ echo 'bf35cb9cdaa340a18f19fa2fc420370b1827ad07 10.128.0.28:30301@32062,redis-cluster-shard-sjh-0.redis-cluster-shard-sjh-headless.default.svc myself,master - 0 1713520897000 5 connected 1365-5460 12288-13653'
++ awk '{print $2}'
+ node_ip_port_fields=10.128.0.28:30301@32062,redis-cluster-shard-sjh-0.redis-cluster-shard-sjh-headless.default.svc
++ echo 10.128.0.28:30301@32062,redis-cluster-shard-sjh-0.redis-cluster-shard-sjh-headless.default.svc
++ awk -F @ '{print $1}'
+ node_ip_port=10.128.0.28:30301
++ echo 10.128.0.28:30301@32062,redis-cluster-shard-sjh-0.redis-cluster-shard-sjh-headless.default.svc
++ awk -F , '{print $1}'
++ awk -F @ '{print $2}'
+ node_bus_port=32062
++ echo 10.128.0.28:30301@32062,redis-cluster-shard-sjh-0.redis-cluster-shard-sjh-headless.default.svc
++ awk -F @ '{print $1}'
++ cut -d: -f2
+ node_port=30301
++ echo 'bf35cb9cdaa340a18f19fa2fc420370b1827ad07 10.128.0.28:30301@32062,redis-cluster-shard-sjh-0.redis-cluster-shard-sjh-headless.default.svc myself,master - 0 1713520897000 5 connected 1365-5460 12288-13653'
++ awk '{print $2}'
++ awk -F , '{print $2}'
+ node_fqdn=redis-cluster-shard-sjh-0.redis-cluster-shard-sjh-headless.default.svc
++ echo 'bf35cb9cdaa340a18f19fa2fc420370b1827ad07 10.128.0.28:30301@32062,redis-cluster-shard-sjh-0.redis-cluster-shard-sjh-headless.default.svc myself,master - 0 1713520897000 5 connected 1365-5460 12288-13653'
++ awk '{print $3}'
+ node_role=myself,master
+ true
+ [[ -n _ ]]
+ [[ myself,master =~ master ]]
+ current_comp_primary_node+=("$node_ip_port@$node_bus_port")
+ read -r line
++ echo 'f5cce503559018a68ffb8f29e2a1ad83f74e9690 10.128.0.30:31843@31771,redis-cluster-shard-sjh-1.redis-cluster-shard-sjh-headless.default.svc slave,fail bf35cb9cdaa340a18f19fa2fc420370b1827ad07 1713520867793 1713520865783 5 connected'
++ awk '{print $2}'
+ node_ip_port_fields=10.128.0.30:31843@31771,redis-cluster-shard-sjh-1.redis-cluster-shard-sjh-headless.default.svc
++ echo 10.128.0.30:31843@31771,redis-cluster-shard-sjh-1.redis-cluster-shard-sjh-headless.default.svc
++ awk -F @ '{print $1}'
+ node_ip_port=10.128.0.30:31843
++ echo 10.128.0.30:31843@31771,redis-cluster-shard-sjh-1.redis-cluster-shard-sjh-headless.default.svc
++ awk -F @ '{print $2}'
++ awk -F , '{print $1}'
+ node_bus_port=31771
++ echo 10.128.0.30:31843@31771,redis-cluster-shard-sjh-1.redis-cluster-shard-sjh-headless.default.svc
++ awk -F @ '{print $1}'
++ cut -d: -f2
+ node_port=31843
++ echo 'f5cce503559018a68ffb8f29e2a1ad83f74e9690 10.128.0.30:31843@31771,redis-cluster-shard-sjh-1.redis-cluster-shard-sjh-headless.default.svc slave,fail bf35cb9cdaa340a18f19fa2fc420370b1827ad07 1713520867793 1713520865783 5 connected'
++ awk '{print $2}'
++ awk -F , '{print $2}'
+ node_fqdn=redis-cluster-shard-sjh-1.redis-cluster-shard-sjh-headless.default.svc
++ echo 'f5cce503559018a68ffb8f29e2a1ad83f74e9690 10.128.0.30:31843@31771,redis-cluster-shard-sjh-1.redis-cluster-shard-sjh-headless.default.svc slave,fail bf35cb9cdaa340a18f19fa2fc420370b1827ad07 1713520867793 1713520865783 5 connected'
++ awk '{print $3}'
+ node_role=slave,fail
+ true
+ [[ -n _ ]]
+ [[ slave,fail =~ master ]]
+ current_comp_other_nodes+=("$node_ip_port@$node_bus_port")
+ read -r line
++ echo '9315a33085454e972d2dd489fffa31900d6a9f47 10.128.0.24:30628@31979,redis-cluster-shard-49r-1.redis-cluster-shard-49r-headless.default.svc master - 0 1713520897000 7 connected'
++ awk '{print $2}'
+ node_ip_port_fields=10.128.0.24:30628@31979,redis-cluster-shard-49r-1.redis-cluster-shard-49r-headless.default.svc
++ echo 10.128.0.24:30628@31979,redis-cluster-shard-49r-1.redis-cluster-shard-49r-headless.default.svc
++ awk -F @ '{print $1}'
+ node_ip_port=10.128.0.24:30628
++ echo 10.128.0.24:30628@31979,redis-cluster-shard-49r-1.redis-cluster-shard-49r-headless.default.svc
++ awk -F @ '{print $2}'
++ awk -F , '{print $1}'
+ node_bus_port=31979
++ echo 10.128.0.24:30628@31979,redis-cluster-shard-49r-1.redis-cluster-shard-49r-headless.default.svc
++ awk -F @ '{print $1}'
++ cut -d: -f2
+ node_port=30628
++ echo '9315a33085454e972d2dd489fffa31900d6a9f47 10.128.0.24:30628@31979,redis-cluster-shard-49r-1.redis-cluster-shard-49r-headless.default.svc master - 0 1713520897000 7 connected'
++ awk '{print $2}'
++ awk -F , '{print $2}'
+ node_fqdn=redis-cluster-shard-49r-1.redis-cluster-shard-49r-headless.default.svc
++ echo '9315a33085454e972d2dd489fffa31900d6a9f47 10.128.0.24:30628@31979,redis-cluster-shard-49r-1.redis-cluster-shard-49r-headless.default.svc master - 0 1713520897000 7 connected'
++ awk '{print $3}'
+ node_role=master
+ true
+ [[ -n '' ]]
+ read -r line
++ echo '88780f6583930a9b4a6f74f1d07d1187af3e56a9 10.128.0.30:30924@31891,redis-cluster-shard-49r-0.redis-cluster-shard-49r-headless.default.svc master - 0 1713520897955 7 connected 6827-10922 15019-16383'
++ awk '{print $2}'
+ node_ip_port_fields=10.128.0.30:30924@31891,redis-cluster-shard-49r-0.redis-cluster-shard-49r-headless.default.svc
++ echo 10.128.0.30:30924@31891,redis-cluster-shard-49r-0.redis-cluster-shard-49r-headless.default.svc
++ awk -F @ '{print $1}'
+ node_ip_port=10.128.0.30:30924
++ echo 10.128.0.30:30924@31891,redis-cluster-shard-49r-0.redis-cluster-shard-49r-headless.default.svc
++ awk -F @ '{print $2}'
++ awk -F , '{print $1}'
+ node_bus_port=31891
++ echo 10.128.0.30:30924@31891,redis-cluster-shard-49r-0.redis-cluster-shard-49r-headless.default.svc
++ awk -F @ '{print $1}'
++ cut -d: -f2
+ node_port=30924
++ echo '88780f6583930a9b4a6f74f1d07d1187af3e56a9 10.128.0.30:30924@31891,redis-cluster-shard-49r-0.redis-cluster-shard-49r-headless.default.svc master - 0 1713520897955 7 connected 6827-10922 15019-16383'
++ awk '{print $2}'
++ awk -F , '{print $2}'
+ node_fqdn=redis-cluster-shard-49r-0.redis-cluster-shard-49r-headless.default.svc
++ echo '88780f6583930a9b4a6f74f1d07d1187af3e56a9 10.128.0.30:30924@31891,redis-cluster-shard-49r-0.redis-cluster-shard-49r-headless.default.svc master - 0 1713520897955 7 connected 6827-10922 15019-16383'
++ awk '{print $3}'
+ node_role=master
+ true
+ [[ -n '' ]]
+ read -r line
++ echo '8126a4ea2434ab9326e221a38b3e16e14d0dad86 10.128.0.30:31022@31002,redis-cluster-shard-wpv-1.redis-cluster-shard-wpv-headless.default.svc slave 42357755c4a03bb37f759348dfdd132738b6a478 1713520893935 1713520891523 6 connected'
++ awk '{print $2}'
+ node_ip_port_fields=10.128.0.30:31022@31002,redis-cluster-shard-wpv-1.redis-cluster-shard-wpv-headless.default.svc
++ echo 10.128.0.30:31022@31002,redis-cluster-shard-wpv-1.redis-cluster-shard-wpv-headless.default.svc
++ awk -F @ '{print $1}'
+ node_ip_port=10.128.0.30:31022
++ echo 10.128.0.30:31022@31002,redis-cluster-shard-wpv-1.redis-cluster-shard-wpv-headless.default.svc
++ awk -F @ '{print $2}'
++ awk -F , '{print $1}'
+ node_bus_port=31002
++ echo 10.128.0.30:31022@31002,redis-cluster-shard-wpv-1.redis-cluster-shard-wpv-headless.default.svc
++ awk -F @ '{print $1}'
++ cut -d: -f2
+ node_port=31022
++ echo '8126a4ea2434ab9326e221a38b3e16e14d0dad86 10.128.0.30:31022@31002,redis-cluster-shard-wpv-1.redis-cluster-shard-wpv-headless.default.svc slave 42357755c4a03bb37f759348dfdd132738b6a478 1713520893935 1713520891523 6 connected'
++ awk '{print $2}'
++ awk -F , '{print $2}'
+ node_fqdn=redis-cluster-shard-wpv-1.redis-cluster-shard-wpv-headless.default.svc
++ echo '8126a4ea2434ab9326e221a38b3e16e14d0dad86 10.128.0.30:31022@31002,redis-cluster-shard-wpv-1.redis-cluster-shard-wpv-headless.default.svc slave 42357755c4a03bb37f759348dfdd132738b6a478 1713520893935 1713520891523 6 connected'
++ awk '{print $3}'
+ node_role=slave
+ true
+ [[ -n '' ]]
+ read -r line
+ echo 'current_comp_primary_node: 10.128.0.28:30301@32062'
current_comp_primary_node: 10.128.0.28:30301@32062
current_comp_other_nodes: 10.128.0.30:31843@31771
+ echo 'current_comp_other_nodes: 10.128.0.30:31843@31771'
+ '[' 1 -eq 0 ']'
+ primary_node_info=10.128.0.28:30301@32062
++ echo 10.128.0.28:30301@32062
++ awk -F @ '{print $1}'
+ primary_node_endpoint_with_port=10.128.0.28:30301
++ echo 10.128.0.28:30301@32062
++ awk -F @ '{print $1}'
++ awk -F : '{print $1}'
+ primary_node_endpoint=10.128.0.28
++ echo 10.128.0.28:30301@32062
++ awk -F @ '{print $1}'
++ awk -F : '{print $2}'
+ primary_node_port=30301
++ echo 10.128.0.28:30301@32062
++ awk -F @ '{print $2}'
+ primary_node_bus_port=32062
+ is_node_in_cluster 10.128.0.28 30301 redis-cluster-shard-sjh-1
+ local random_node_endpoint=10.128.0.28
+ local random_node_port=30301
+ local node_name=redis-cluster-shard-sjh-1
+ '[' -z O3605v7HsS ']'
++ redis-cli -h 10.128.0.28 -p 30301 -a O3605v7HsS cluster nodes
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
+ cluster_nodes_info='42357755c4a03bb37f759348dfdd132738b6a478 10.128.0.28:32745@32317,redis-cluster-shard-wpv-0.redis-cluster-shard-wpv-headless.default.svc master - 0 1713520896949 6 connected 0-1364 5461-6826 10923-12287 13654-15018
bf35cb9cdaa340a18f19fa2fc420370b1827ad07 10.128.0.28:30301@32062,redis-cluster-shard-sjh-0.redis-cluster-shard-sjh-headless.default.svc myself,master - 0 1713520897000 5 connected 1365-5460 12288-13653
f5cce503559018a68ffb8f29e2a1ad83f74e9690 10.128.0.30:31843@31771,redis-cluster-shard-sjh-1.redis-cluster-shard-sjh-headless.default.svc slave,fail bf35cb9cdaa340a18f19fa2fc420370b1827ad07 1713520867793 1713520865783 5 connected
9315a33085454e972d2dd489fffa31900d6a9f47 10.128.0.24:30628@31979,redis-cluster-shard-49r-1.redis-cluster-shard-49r-headless.default.svc master - 0 1713520897000 7 connected
88780f6583930a9b4a6f74f1d07d1187af3e56a9 10.128.0.30:30924@31891,redis-cluster-shard-49r-0.redis-cluster-shard-49r-headless.default.svc master - 0 1713520897955 7 connected 6827-10922 15019-16383
8126a4ea2434ab9326e221a38b3e16e14d0dad86 10.128.0.30:31022@31002,redis-cluster-shard-wpv-1.redis-cluster-shard-wpv-headless.default.svc slave 42357755c4a03bb37f759348dfdd132738b6a478 1713520893935 1713520891523 6 connected'
++ echo '42357755c4a03bb37f759348dfdd132738b6a478 10.128.0.28:32745@32317,redis-cluster-shard-wpv-0.redis-cluster-shard-wpv-headless.default.svc master - 0 1713520896949 6 connected 0-1364 5461-6826 10923-12287 13654-15018
bf35cb9cdaa340a18f19fa2fc420370b1827ad07 10.128.0.28:30301@32062,redis-cluster-shard-sjh-0.redis-cluster-shard-sjh-headless.default.svc myself,master - 0 1713520897000 5 connected 1365-5460 12288-13653
f5cce503559018a68ffb8f29e2a1ad83f74e9690 10.128.0.30:31843@31771,redis-cluster-shard-sjh-1.redis-cluster-shard-sjh-headless.default.svc slave,fail bf35cb9cdaa340a18f19fa2fc420370b1827ad07 1713520867793 1713520865783 5 connected
9315a33085454e972d2dd489fffa31900d6a9f47 10.128.0.24:30628@31979,redis-cluster-shard-49r-1.redis-cluster-shard-49r-headless.default.svc master - 0 1713520897000 7 connected
88780f6583930a9b4a6f74f1d07d1187af3e56a9 10.128.0.30:30924@31891,redis-cluster-shard-49r-0.redis-cluster-shard-49r-headless.default.svc master - 0 1713520897955 7 connected 6827-10922 15019-16383
8126a4ea2434ab9326e221a38b3e16e14d0dad86 10.128.0.30:31022@31002,redis-cluster-shard-wpv-1.redis-cluster-shard-wpv-headless.default.svc slave 42357755c4a03bb37f759348dfdd132738b6a478 1713520893935 1713520891523 6 connected'
++ wc -l
+ '[' 6 -gt 1 ']'
+ echo '42357755c4a03bb37f759348dfdd132738b6a478 10.128.0.28:32745@32317,redis-cluster-shard-wpv-0.redis-cluster-shard-wpv-headless.default.svc master - 0 1713520896949 6 connected 0-1364 5461-6826 10923-12287 13654-15018
bf35cb9cdaa340a18f19fa2fc420370b1827ad07 10.128.0.28:30301@32062,redis-cluster-shard-sjh-0.redis-cluster-shard-sjh-headless.default.svc myself,master - 0 1713520897000 5 connected 1365-5460 12288-13653
f5cce503559018a68ffb8f29e2a1ad83f74e9690 10.128.0.30:31843@31771,redis-cluster-shard-sjh-1.redis-cluster-shard-sjh-headless.default.svc slave,fail bf35cb9cdaa340a18f19fa2fc420370b1827ad07 1713520867793 1713520865783 5 connected
9315a33085454e972d2dd489fffa31900d6a9f47 10.128.0.24:30628@31979,redis-cluster-shard-49r-1.redis-cluster-shard-49r-headless.default.svc master - 0 1713520897000 7 connected
88780f6583930a9b4a6f74f1d07d1187af3e56a9 10.128.0.30:30924@31891,redis-cluster-shard-49r-0.redis-cluster-shard-49r-headless.default.svc master - 0 1713520897955 7 connected 6827-10922 15019-16383
8126a4ea2434ab9326e221a38b3e16e14d0dad86 10.128.0.30:31022@31002,redis-cluster-shard-wpv-1.redis-cluster-shard-wpv-headless.default.svc slave 42357755c4a03bb37f759348dfdd132738b6a478 1713520893935 1713520891523 6 connected'
+ grep -q redis-cluster-shard-sjh-1
Node redis-cluster-shard-sjh-1 is already in the cluster, skipping...
+ true
+ echo 'Node redis-cluster-shard-sjh-1 is already in the cluster, skipping...'
+ exit 0
@ahjing99 ahjing99 added the kind/bug Something isn't working label Apr 19, 2024
@ahjing99 ahjing99 added this to the Release 0.9.0 milestone Apr 19, 2024
@ahjing99 ahjing99 added the severity/major Great chance user will encounter the same problem label Apr 19, 2024
@ahjing99 ahjing99 removed the severity/major Great chance user will encounter the same problem label May 6, 2024
@ahjing99
Copy link
Collaborator Author

ahjing99 commented May 6, 2024

cannot recreate on 0.9.0-beta.17

Copy link

This issue has been marked as stale because it has been open for 30 days with no activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Something isn't working Stale
Projects
None yet
Development

No branches or pull requests

2 participants