[BUG] Elevated Latency in pipeline SET and MGET calls with version 9.5.2 or higher

[tim-zhu-reddit]

Expected Behavior

We use redis pipelining (https://pkg.go.dev/github.com/go-redis/redis#Pipeline.Exec) to execute a series of SET commands.
We expected the Pipeline.Exec() call to take less than 200ms at p99, 2ms at p50
We use MGET(https://pkg.go.dev/github.com/go-redis/redis#Ring.MGet) to get keys from redis, expect latency at 1ms p50, 100ms p99 at peak times.

Current Behavior

With Version 9.5.2(https://github.com/redis/go-redis/releases/tag/v9.5.2) and higher, P99 latency spiked above 4s for pipeline SET, causing timeout. and 2s for MGET, while p50 latency remains steady under 2ms and 1ms respectively.
We first noticed the issue after upgrading the version from 9.2.0 to 9.7.0. We bisect-tested different versions and determined 9.5.2 was the first bad version with increased latency.

Possible Solution

Steps to Reproduce

1. change version to 9.5.2 or higher
2. deploy service as usual
3. observe p99 latency jumps immediately for pipeline SET and MGET requests

Context (Environment)

Our use redis as a simple dudupe layer, acting like a LRU cache. When we see a new name, we write to Redis, then check redis to see if we have seen this name.
peak hours QPS are 350k for MGET and 25k for pipeline request(max 25 SET command in each pipeline request)

This latency issue seem to impact high QPS service like this, as we didn't hear other internal teams using the same go-redis version reporting a similar issue.

Detailed Description

Possible Implementation

[ndyakov]

@tim-zhu-reddit thank you for reporting this. Have any changes to the redis server itself happened while this latency increase was observed? Was the change in latency solely based on the change of go-redis version?

[ndyakov]

Also, if you are correct that the bug is present in v9.5.2 but not in v.9.5.1 the only change around the pipeline code that I tracked and can start investigating is this one: #2961

[tim-zhu-reddit]

Have any changes to the redis server itself happened while this latency increase was observed? Was the change in latency solely based on the change of go-redis version?

@ndyakov thank you for taking a look!
we didn't have any server changes. and since we could reproduce the issue consistently with 9.5.2 or higher and earlier versions consistently working, we think it is due some a change in v9.5.2

[fishy]

@ndyakov : we started to notice the issue when we upgraded client lib from v9.2.0 to v9.7.0.

after noticed the issue, we did a "bisect" on the releases between them, and found out that v9.5.1 is the last good one and v9.5.2 is the first bad one.

if you think #2961 is the only one could be relevant in 9.5.2 release, then we can try testing 9.7.0 with revert of that one applied (assuming it's still a clean revert) to see if we can reproduce the bug.

[fishy]

Update: We tested 9.7.0 with #2961 reverted (see https://github.com/fishy/go-redis/commits/38b8f52b4d563cb0c07557c254e4a9b16d25674e/), and verified that version does not have the same issue.

So the bug should be somewhere inside #2961.

[ndyakov]

@fishy: that is interesting. Will look into it, see if I can prepare benchmark test that will catch the degradation of performance and see how to proceed. I expect this to take some time, in the meantime I hope the approach with the reverted PR works for you.

By the way, do you have timeouts on the context that you are passing or are you canceling the context by any chance ?

[ndyakov]

@fishy if you are using cancellation of the context, would you please test against #3190 to see if the issue persists?

[fishy]

@ndyakov we tested and #3190 does not fix the issue we are seeing, the latency is still high with that version.

[ndyakov]

Understood. @fishy are you observing something additional in the node logs or go-redis logs? I assume we should add more logging around this piece of code with the markNodeAsFailing in the said PR to observe which flow is introducing the increased latency. Do you think we can work together on a benchmark test that reflects your usage so we can catch the difference in performance?

[tim-zhu-reddit]

@ndyakov I looked into the server-side metrics for two tests we ran with the new version yesterday.
it seems the elevated latency may have come from the read requests

• I see a change in read traffic and latency, while set latency remain consistant
• some redis nodes that used to get 0 requests suddenly had read traffic, while previous higher traffic nodes had their requests dropped abit. it almost looked like it became a more even request rate among nodes during the high latency period.

below is a screenshot of the read and write request rate and latency by redis nodes

on https://pkg.go.dev/github.com/go-redis/redis#ClusterOptions, we set readonly = true and routeRandomly = true. could #2961 cause a behaviour change in routing read requests?

[ndyakov]

Hm.. that is quite strange that you are observing. It is interesting to see more equal distribution but higher latency. I am still focused on another feature, but would like to dig deeper into the issue since we have pinpointed the exact patch that triggers it.

It looks like the package is marking the nodes as unhealthy and rerouting requests to other nodes. What is quite strange for me is that during normal periods it doesn't seem that you have equal distribution of read requests. Maybe there is something in the application code that is reusing a sticky connection? (cc @tim-zhu-reddit , @fishy )

[EXPEbdodla]

@ndyakov Server side metrics look similar to #3190 PR. It may be the case when error (may be context error) happens in pipelineReadCmds method and isBadConnection() is marking node as failed. May be @fishy or @tim-zhu-reddit can share more information about the client configuration options and timeouts set.

[tim-zhu-reddit]

@EXPEbdodla thank you for helping, here is our config
ReadOnly - true
RouteRandomly - true
ReadTimeout: 3s
DialTimeout: 3s
WriteTimeout: 3s
other https://pkg.go.dev/github.com/go-redis/redis#ClusterOptions are unset

[tim-zhu-reddit]

Maybe there is something in the application code that is reusing a sticky connection?

@ndyakov, good question. I double check our code, we don't have any logic around sticky connections. since we set RouteRandomly = true in the ClusterOption.

[monkey92t]

I reviewed the code in #2961, and it only handles ERR. I noticed that you confirmed the issue was alleviated after reverting to the #2961 code, which suggests that in your example, there were a large number of request errors (possibly Redis errors or network errors).

I suspect it might be a MOVED or READONLY error. If you encountered a network error, reverting to #2961 wouldn’t improve the situation.

Next, let’s assume it’s a MOVED error. There are two scenarios where this error occurs: either the slot calculation for the Redis cluster was incorrect, but I checked the osscluster.go code and found no issues; or the cluster slots changed, and go-redis synchronizes this lazily. In your test, I believe this didn’t happen, as your tests were completed in a very short time frame.

Finally, there’s the READONLY error, but I still can’t pinpoint where it goes wrong. In the mapCmdsByNode function, it only uses slave nodes when all commands in the pipeline are read-only; otherwise, it uses master nodes exclusively. This approach doesn’t seem to cause any errors, as it wouldn’t send write commands to a read-only node.

We must be missing something. If possible, could you help identify where the error occurs? For example, is it a ctx, net, MOVED, or READONLY error?

Additionally, could you provide the test code you used? This would help reproduce the issue.

[monkey92t]

Alternatively, you could use the process of elimination by reverting the code from #2961 line by line, which could help pinpoint the location of the issue.

[tim-zhu-reddit]

@monkey92t reverting the last line of change in #2961 fixed the issue for us. we also tested latest master, confirmed that #is still problematic. so it is very likely it is that line caused our issue.

[monkey92t]

@monkey92t reverting the last line of change in #2961 fixed the issue for us. we also tested latest master, confirmed that #is still problematic. so it is very likely it is that line caused our issue.

Thank you for your help. Now we can almost confirm the existence of a Redis-ERROR, and it might even be triggered by ReadOnly. We need to conduct further troubleshooting:


1. If you try setting opt.ReadOnly to false, will the situation improve?

2. Can you provide details about which commands were used in the pipeline? For example, get, set, hdel, etc.

3. Can you provide your cluster status? The output of the cluster nodes and cluster slots commands.

From the symptoms, it seems to be a READONLY error. You can confirm this in the isBadConn function. However, what’s puzzling is that in cluster mode, write commands should not be sent to ReadOnly nodes. If possible, could you provide a reproducible example?

[fishy]

If possible, could you provide a reproducible example?

unfortunately we cannot even reproduce it on our staging environment, so we have to use our production to verify whether the different versions of go-redis have or don't have the bug (when they do have the bug, the consequence is mild, fortunately)

tim will update on other questions.

[tim-zhu-reddit]

I tested it out with ReadOnly and RouteRandomly set to false, still having the issue. I ran CLUSTER NODES and CLUSTER SLOTS in the cluster, the only thing I noticed is that three nodes are disconnected, disconnected 3164-3275 11196-11467 15292-15563. I also did the same with the good version (one-line reverted)

Problematic version (readonly turned off)

redis-xxxxx.net:6379> CLUSTER NODES
6ca0e6e793c2d73601e566017b3920ce0a032c99 10.24.96.34:6379@16379 slave d8601f4f05eb728c718cb2c696a6e0dd1d861f64 0 1741724609538 38 connected
3eaf634b52441c9e6a31398e50e55e7c1112ae84 10.24.95.63:6379@16379 master - 0 1741724609000 25 connected 711-1364 3003
dc9e0b400639917294259b5ac25cc469656b7797 10.24.105.11:6379@16379 slave cb0e71df63c9081af3245c86ebee537e2e8237e4 0 1741724610041 8 connected
565734a5485c850517eda56d7076ec2b3a6a9d65 10.24.73.189:6379@16379 master - 0 1741724609538 7 connected 8902-9556
075e9f0a1a53569f6c8e6a4821937e614bd12ee9 10.24.96.124:6379@16379 master - 0 1741724610041 21 connected 4258-4368 8192-8314 8316-8463 15019-15291
3c3afe5f7c89560c12c59ab763a8ea47107b9fa6 10.24.91.225:6379@16379 slave 29728be273e78753f3b7ad2fefad98b42f6fcec3 0 1741724610543 10 connected
bcb18281cbdac6c1c514cc2d2a89cd2a27efba8a 10.24.82.77:6379@16379 slave 96c477de9f8f766c8d5ad8e1870df7adac2322e3 0 1741724609537 36 connected
f5a9cfd1a66cc797a32b583efb3da782b45da02a 10.24.109.70:6379@16379 slave 0be8e0a01cd11c770140e5e1aacbc7e11f2047b9 0 1741724610543 55 connected
587523f1d707f9186941f7e1e7f9ed39345d0612 10.24.73.3:6379@16379 slave 3720b2c0cded538c9fdbad851a40bd02b58358ed 0 1741724610040 23 connected
96c477de9f8f766c8d5ad8e1870df7adac2322e3 10.24.70.114:6379@16379 master - 0 1741724609302 36 connected 4533-4641 8465-8737 12561-12833
c450835ccc66843c44d9768f390d860fc9633a6b 10.24.106.131:6379@16379 master - 1741724608230 1741724606000 35 disconnected 3164-3275 11196-11467 15292-15563
87b0b8ad2a7eafc41d80163b29658efd321ff089 10.24.106.170:6379@16379 master - 0 1741724609798 51 connected 273 1365-1366 1639 3004-3163 3276-3439 11468-11631 15564-15727
1c72a09c55fef2a5a05bd40fac7ffeb8aeb5b995 10.24.89.51:6379@16379 master - 0 1741724610543 5 connected 6172-6826
fba74bd3be58e93ccf328914a57407bc68b677af 10.24.72.35:6379@16379 slave 075e9f0a1a53569f6c8e6a4821937e614bd12ee9 0 1741724610543 21 connected
16919de207c198061ededb37c8590532239828ef 10.24.75.131:6379@16379 master - 0 1741724610543 20 connected 6992-7099 8315 10923-11195 12288-12560
d8601f4f05eb728c718cb2c696a6e0dd1d861f64 10.24.82.226:6379@16379 master - 0 1741724610040 38 connected 1804-1912 5735-6007 13927-14199
b95f503223b92cfb109e4561399339b5702f1beb 10.24.90.220:6379@16379 myself,master - 0 1741724609000 22 connected 2077-2730 8464
4855173a8e10c9209eedbe101de8a36aa8692fbb 10.24.94.166:6379@16379 master - 0 1741724610543 39 connected 164-272 274-546 7100-7372
ffead3ee63fab8831959124a5eee93da75397b7a 10.24.92.116:6379@16379 slave c92b87eec32eaf7da0aa73fdf5694c3c5d23f59d 0 1741724609537 24 connected
d5248d7b96f941b8931544aebdfc6cfcfdd289d9 10.24.70.32:6379@16379 slave 9524fc5ffbc0e44651c0f96fe8f8f842fdc10bd0 0 1741724610000 26 connected
9da853291de8f09bb932e8e09a6b48d810a1c962 10.24.108.1:6379@16379 master - 0 1741724610544 52 connected 1367-1369 4369-4532 4642-4804 7373-7535 12834-12996
a902832eb3b9cc78c95da2e383c0335058835ed7 10.24.109.201:6379@16379 slave 4855173a8e10c9209eedbe101de8a36aa8692fbb 0 1741724609000 39 connected
152adf2b32817c09fae6cd259bdcab01f9584c35 10.24.96.147:6379@16379 slave 80128f306c2fa8711ce25b3dfa125152e4310d92 0 1741724609538 42 connected
96914b07338c13f77a3828cdc355398bd164db10 10.24.85.111:6379@16379 slave 5d3c84ccbb22a6b0457876b79ce118e0e527e6b3 0 1741724608532 37 connected
de3e35b1b2e91d1ed503bbfa26bcb20608989453 10.24.92.75:6379@16379 master - 0 1741724610543 11 connected 14364-15018
2776b5be74a79da05f872ab88987ec3672c5f004 10.24.79.174:6379@16379 slave c28db2886b5e05c9b36b28da03ac54ffc2e3dee3 0 1741724609000 27 connected
c186b182cdbbc306f97ea786f110f9fd92110568 10.24.98.0:6379@16379 slave 1bcebfbdcb02cbf33f219c9f2b1b9a072a9b401c 0 1741724610000 18 connected
df17ee6c3aa5aa4d6cd84cbf640737d30d265dc0 10.24.79.25:6379@16379 master - 1741724610241 1741724608030 54 connected 1642-1803 1913-2076 5464-5627 6008-6171 14200
7496fa3a15564b45d603265fb6c8e0444c782738 10.24.65.183:6379@16379 slave 9da853291de8f09bb932e8e09a6b48d810a1c962 0 1741724609538 52 connected
e81222987fa225e444e64306ad33d30adfffb5ec 10.24.107.148:6379@16379 slave b95f503223b92cfb109e4561399339b5702f1beb 0 1741724610000 22 connected
cb0e71df63c9081af3245c86ebee537e2e8237e4 10.24.81.202:6379@16379 master - 0 1741724610285 8 connected 10268-10922
c28db2886b5e05c9b36b28da03ac54ffc2e3dee3 10.24.102.0:6379@16379 master - 0 1741724609538 27 connected 11632-12287
64ab49d38b0a7bf691c28a62d4cf1258b8c4b73d 10.24.98.235:6379@16379 slave de3e35b1b2e91d1ed503bbfa26bcb20608989453 0 1741724610000 11 connected
c92b87eec32eaf7da0aa73fdf5694c3c5d23f59d 10.24.66.212:6379@16379 master - 0 1741724609538 24 connected 4805-5460
1bcebfbdcb02cbf33f219c9f2b1b9a072a9b401c 10.24.88.141:6379@16379 master - 0 1741724610543 18 connected 5628-5734 9557-9830 13653-13926
5d3c84ccbb22a6b0457876b79ce118e0e527e6b3 10.24.72.95:6379@16379 master - 0 1741724609538 37 connected 1531-1636 2731-3002 5461-5463 6827 9831-10103
4e1a822779e4b2b68cb8d4584899d477806c5339 10.24.72.90:6379@16379 slave 87b0b8ad2a7eafc41d80163b29658efd321ff089 0 1741724609538 51 connected
80128f306c2fa8711ce25b3dfa125152e4310d92 10.24.67.179:6379@16379 master - 0 1741724610000 42 connected 7536-8191
29728be273e78753f3b7ad2fefad98b42f6fcec3 10.24.67.254:6379@16379 master - 0 1741724609000 10 connected 12997-13652
df68a94dc2a71853ad591e93c1c4405f30455216 10.24.82.36:6379@16379 slave 16919de207c198061ededb37c8590532239828ef 0 1741724610040 20 connected
3720b2c0cded538c9fdbad851a40bd02b58358ed 10.24.110.223:6379@16379 master - 0 1741724610000 23 connected 15728-16383
9524fc5ffbc0e44651c0f96fe8f8f842fdc10bd0 10.24.100.82:6379@16379 master - 0 1741724610757 26 connected 3440-4095
873a694cdb740c71e988a6240f518eec00080f36 10.24.66.51:6379@16379 slave 3eaf634b52441c9e6a31398e50e55e7c1112ae84 0 1741724610141 25 connected
0be8e0a01cd11c770140e5e1aacbc7e11f2047b9 10.24.87.22:6379@16379 master - 0 1741724609537 55 connected 0-163 547-710 1637-1638 4096-4257 14201-14363
f635a537362533d04e51d0d9dc3d3af840ca1280 10.24.70.53:6379@16379 slave c450835ccc66843c44d9768f390d860fc9633a6b 0 1741724610040 35 connected
0c58d020fef0f4a416094838c0c54ee59ee83eb4 10.24.80.142:6379@16379 slave 7924491090d15e1d9405b4e84fa16b5bdd62106f 0 1741724609537 53 connected
d34559353ce7d57f3d7f09041e414f659b32b480 10.24.82.90:6379@16379 slave df17ee6c3aa5aa4d6cd84cbf640737d30d265dc0 0 1741724610543 54 connected
7924491090d15e1d9405b4e84fa16b5bdd62106f 10.24.69.144:6379@16379 master - 0 1741724610544 53 connected 1370-1530 1640-1641 6828-6991 8738-8901 10104-10267
c8ffe91c3a493ec29d546923e0eab25bd59622f8 10.24.80.229:6379@16379 slave 565734a5485c850517eda56d7076ec2b3a6a9d65 0 1741724609000 7 connected
4b4a74b1ed6ef93a84ddda7c4f3a609335ccf125 10.24.108.244:6379@16379 slave 1c72a09c55fef2a5a05bd40fac7ffeb8aeb5b995 0 1741724610543 5 connected
redis-xxxxx.net:6379> CLUSTER SLOTS
 1) 1) (integer) 0
    2) (integer) 163
    3) 1) "10.24.87.22"
       2) (integer) 6379
       3) "0be8e0a01cd11c770140e5e1aacbc7e11f2047b9"
    4) 1) "10.24.109.70"
       2) (integer) 6379
       3) "f5a9cfd1a66cc797a32b583efb3da782b45da02a"
 2) 1) (integer) 164
    2) (integer) 272
    3) 1) "10.24.94.166"
       2) (integer) 6379
       3) "4855173a8e10c9209eedbe101de8a36aa8692fbb"
    4) 1) "10.24.109.201"
       2) (integer) 6379
       3) "a902832eb3b9cc78c95da2e383c0335058835ed7"
 3) 1) (integer) 273
    2) (integer) 273
    3) 1) "10.24.106.170"
       2) (integer) 6379
       3) "87b0b8ad2a7eafc41d80163b29658efd321ff089"
    4) 1) "10.24.72.90"
       2) (integer) 6379
       3) "4e1a822779e4b2b68cb8d4584899d477806c5339"
 4) 1) (integer) 274
    2) (integer) 546
    3) 1) "10.24.94.166"
       2) (integer) 6379
       3) "4855173a8e10c9209eedbe101de8a36aa8692fbb"
    4) 1) "10.24.109.201"
       2) (integer) 6379
       3) "a902832eb3b9cc78c95da2e383c0335058835ed7"
 5) 1) (integer) 547
    2) (integer) 710
    3) 1) "10.24.87.22"
       2) (integer) 6379
       3) "0be8e0a01cd11c770140e5e1aacbc7e11f2047b9"
    4) 1) "10.24.109.70"
       2) (integer) 6379
       3) "f5a9cfd1a66cc797a32b583efb3da782b45da02a"
 6) 1) (integer) 711
    2) (integer) 1364
    3) 1) "10.24.95.63"
       2) (integer) 6379
       3) "3eaf634b52441c9e6a31398e50e55e7c1112ae84"
    4) 1) "10.24.66.51"
       2) (integer) 6379
       3) "873a694cdb740c71e988a6240f518eec00080f36"
 7) 1) (integer) 1365
    2) (integer) 1366
    3) 1) "10.24.106.170"
       2) (integer) 6379
       3) "87b0b8ad2a7eafc41d80163b29658efd321ff089"
    4) 1) "10.24.72.90"
       2) (integer) 6379
       3) "4e1a822779e4b2b68cb8d4584899d477806c5339"
 8) 1) (integer) 1367
    2) (integer) 1369
    3) 1) "10.24.108.1"
       2) (integer) 6379
       3) "9da853291de8f09bb932e8e09a6b48d810a1c962"
    4) 1) "10.24.65.183"
       2) (integer) 6379
       3) "7496fa3a15564b45d603265fb6c8e0444c782738"
 9) 1) (integer) 1370
    2) (integer) 1530
    3) 1) "10.24.69.144"
       2) (integer) 6379
       3) "7924491090d15e1d9405b4e84fa16b5bdd62106f"
    4) 1) "10.24.80.142"
       2) (integer) 6379
       3) "0c58d020fef0f4a416094838c0c54ee59ee83eb4"
10) 1) (integer) 1531
    2) (integer) 1636
    3) 1) "10.24.72.95"
       2) (integer) 6379
       3) "5d3c84ccbb22a6b0457876b79ce118e0e527e6b3"
    4) 1) "10.24.85.111"
       2) (integer) 6379
       3) "96914b07338c13f77a3828cdc355398bd164db10"
11) 1) (integer) 1637
    2) (integer) 1638
    3) 1) "10.24.87.22"
       2) (integer) 6379
       3) "0be8e0a01cd11c770140e5e1aacbc7e11f2047b9"
    4) 1) "10.24.109.70"
       2) (integer) 6379
       3) "f5a9cfd1a66cc797a32b583efb3da782b45da02a"
12) 1) (integer) 1639
    2) (integer) 1639
    3) 1) "10.24.106.170"
       2) (integer) 6379
       3) "87b0b8ad2a7eafc41d80163b29658efd321ff089"
    4) 1) "10.24.72.90"
       2) (integer) 6379
       3) "4e1a822779e4b2b68cb8d4584899d477806c5339"
13) 1) (integer) 1640
    2) (integer) 1641
    3) 1) "10.24.69.144"
       2) (integer) 6379
       3) "7924491090d15e1d9405b4e84fa16b5bdd62106f"
    4) 1) "10.24.80.142"
       2) (integer) 6379
       3) "0c58d020fef0f4a416094838c0c54ee59ee83eb4"
14) 1) (integer) 1642
    2) (integer) 1803
    3) 1) "10.24.79.25"
       2) (integer) 6379
       3) "df17ee6c3aa5aa4d6cd84cbf640737d30d265dc0"
    4) 1) "10.24.82.90"
       2) (integer) 6379
       3) "d34559353ce7d57f3d7f09041e414f659b32b480"
15) 1) (integer) 1804
    2) (integer) 1912
    3) 1) "10.24.82.226"
       2) (integer) 6379
       3) "d8601f4f05eb728c718cb2c696a6e0dd1d861f64"
    4) 1) "10.24.96.34"
       2) (integer) 6379
       3) "6ca0e6e793c2d73601e566017b3920ce0a032c99"
16) 1) (integer) 1913
    2) (integer) 2076
    3) 1) "10.24.79.25"
       2) (integer) 6379
       3) "df17ee6c3aa5aa4d6cd84cbf640737d30d265dc0"
    4) 1) "10.24.82.90"
       2) (integer) 6379
       3) "d34559353ce7d57f3d7f09041e414f659b32b480"
17) 1) (integer) 2077
    2) (integer) 2730
    3) 1) "10.24.90.220"
       2) (integer) 6379
       3) "b95f503223b92cfb109e4561399339b5702f1beb"
    4) 1) "10.24.107.148"
       2) (integer) 6379
       3) "e81222987fa225e444e64306ad33d30adfffb5ec"
18) 1) (integer) 2731
    2) (integer) 3002
    3) 1) "10.24.72.95"
       2) (integer) 6379
       3) "5d3c84ccbb22a6b0457876b79ce118e0e527e6b3"
    4) 1) "10.24.85.111"
       2) (integer) 6379
       3) "96914b07338c13f77a3828cdc355398bd164db10"
19) 1) (integer) 3003
    2) (integer) 3003
    3) 1) "10.24.95.63"
       2) (integer) 6379
       3) "3eaf634b52441c9e6a31398e50e55e7c1112ae84"
    4) 1) "10.24.66.51"
       2) (integer) 6379
       3) "873a694cdb740c71e988a6240f518eec00080f36"
20) 1) (integer) 3004
    2) (integer) 3163
    3) 1) "10.24.106.170"
       2) (integer) 6379
       3) "87b0b8ad2a7eafc41d80163b29658efd321ff089"
    4) 1) "10.24.72.90"
       2) (integer) 6379
       3) "4e1a822779e4b2b68cb8d4584899d477806c5339"
21) 1) (integer) 3164
    2) (integer) 3275
    3) 1) "10.24.106.131"
       2) (integer) 6379
       3) "c450835ccc66843c44d9768f390d860fc9633a6b"
    4) 1) "10.24.70.53"
       2) (integer) 6379
       3) "f635a537362533d04e51d0d9dc3d3af840ca1280"
22) 1) (integer) 3276
    2) (integer) 3439
    3) 1) "10.24.106.170"
       2) (integer) 6379
       3) "87b0b8ad2a7eafc41d80163b29658efd321ff089"
    4) 1) "10.24.72.90"
       2) (integer) 6379
       3) "4e1a822779e4b2b68cb8d4584899d477806c5339"
23) 1) (integer) 3440
    2) (integer) 4095
    3) 1) "10.24.100.82"
       2) (integer) 6379
       3) "9524fc5ffbc0e44651c0f96fe8f8f842fdc10bd0"
    4) 1) "10.24.70.32"
       2) (integer) 6379
       3) "d5248d7b96f941b8931544aebdfc6cfcfdd289d9"
24) 1) (integer) 4096
    2) (integer) 4257
    3) 1) "10.24.87.22"
       2) (integer) 6379
       3) "0be8e0a01cd11c770140e5e1aacbc7e11f2047b9"
    4) 1) "10.24.109.70"
       2) (integer) 6379
       3) "f5a9cfd1a66cc797a32b583efb3da782b45da02a"
25) 1) (integer) 4258
    2) (integer) 4368
    3) 1) "10.24.96.124"
       2) (integer) 6379
       3) "075e9f0a1a53569f6c8e6a4821937e614bd12ee9"
    4) 1) "10.24.72.35"
       2) (integer) 6379
       3) "fba74bd3be58e93ccf328914a57407bc68b677af"
26) 1) (integer) 4369
    2) (integer) 4532
    3) 1) "10.24.108.1"
       2) (integer) 6379
       3) "9da853291de8f09bb932e8e09a6b48d810a1c962"
    4) 1) "10.24.65.183"
       2) (integer) 6379
       3) "7496fa3a15564b45d603265fb6c8e0444c782738"
27) 1) (integer) 4533
    2) (integer) 4641
    3) 1) "10.24.70.114"
       2) (integer) 6379
       3) "96c477de9f8f766c8d5ad8e1870df7adac2322e3"
    4) 1) "10.24.82.77"
       2) (integer) 6379
       3) "bcb18281cbdac6c1c514cc2d2a89cd2a27efba8a"
28) 1) (integer) 4642
    2) (integer) 4804
    3) 1) "10.24.108.1"
       2) (integer) 6379
       3) "9da853291de8f09bb932e8e09a6b48d810a1c962"
    4) 1) "10.24.65.183"
       2) (integer) 6379
       3) "7496fa3a15564b45d603265fb6c8e0444c782738"
29) 1) (integer) 4805
    2) (integer) 5460
    3) 1) "10.24.66.212"
       2) (integer) 6379
       3) "c92b87eec32eaf7da0aa73fdf5694c3c5d23f59d"
    4) 1) "10.24.92.116"
       2) (integer) 6379
       3) "ffead3ee63fab8831959124a5eee93da75397b7a"
30) 1) (integer) 5461
    2) (integer) 5463
    3) 1) "10.24.72.95"
       2) (integer) 6379
       3) "5d3c84ccbb22a6b0457876b79ce118e0e527e6b3"
    4) 1) "10.24.85.111"
       2) (integer) 6379
       3) "96914b07338c13f77a3828cdc355398bd164db10"
31) 1) (integer) 5464
    2) (integer) 5627
    3) 1) "10.24.79.25"
       2) (integer) 6379
       3) "df17ee6c3aa5aa4d6cd84cbf640737d30d265dc0"
    4) 1) "10.24.82.90"
       2) (integer) 6379
       3) "d34559353ce7d57f3d7f09041e414f659b32b480"
32) 1) (integer) 5628
    2) (integer) 5734
    3) 1) "10.24.88.141"
       2) (integer) 6379
       3) "1bcebfbdcb02cbf33f219c9f2b1b9a072a9b401c"
    4) 1) "10.24.98.0"
       2) (integer) 6379
       3) "c186b182cdbbc306f97ea786f110f9fd92110568"
33) 1) (integer) 5735
    2) (integer) 6007
    3) 1) "10.24.82.226"
       2) (integer) 6379
       3) "d8601f4f05eb728c718cb2c696a6e0dd1d861f64"
    4) 1) "10.24.96.34"
       2) (integer) 6379
       3) "6ca0e6e793c2d73601e566017b3920ce0a032c99"
34) 1) (integer) 6008
    2) (integer) 6171
    3) 1) "10.24.79.25"
       2) (integer) 6379
       3) "df17ee6c3aa5aa4d6cd84cbf640737d30d265dc0"
    4) 1) "10.24.82.90"
       2) (integer) 6379
       3) "d34559353ce7d57f3d7f09041e414f659b32b480"
35) 1) (integer) 6172
    2) (integer) 6826
    3) 1) "10.24.89.51"
       2) (integer) 6379
       3) "1c72a09c55fef2a5a05bd40fac7ffeb8aeb5b995"
    4) 1) "10.24.108.244"
       2) (integer) 6379
       3) "4b4a74b1ed6ef93a84ddda7c4f3a609335ccf125"
36) 1) (integer) 6827
    2) (integer) 6827
    3) 1) "10.24.72.95"
       2) (integer) 6379
       3) "5d3c84ccbb22a6b0457876b79ce118e0e527e6b3"
    4) 1) "10.24.85.111"
       2) (integer) 6379
       3) "96914b07338c13f77a3828cdc355398bd164db10"
37) 1) (integer) 6828
    2) (integer) 6991
    3) 1) "10.24.69.144"
       2) (integer) 6379
       3) "7924491090d15e1d9405b4e84fa16b5bdd62106f"
    4) 1) "10.24.80.142"
       2) (integer) 6379
       3) "0c58d020fef0f4a416094838c0c54ee59ee83eb4"
38) 1) (integer) 6992
    2) (integer) 7099
    3) 1) "10.24.75.131"
       2) (integer) 6379
       3) "16919de207c198061ededb37c8590532239828ef"
    4) 1) "10.24.82.36"
       2) (integer) 6379
       3) "df68a94dc2a71853ad591e93c1c4405f30455216"
39) 1) (integer) 7100
    2) (integer) 7372
    3) 1) "10.24.94.166"
       2) (integer) 6379
       3) "4855173a8e10c9209eedbe101de8a36aa8692fbb"
    4) 1) "10.24.109.201"
       2) (integer) 6379
       3) "a902832eb3b9cc78c95da2e383c0335058835ed7"
40) 1) (integer) 7373
    2) (integer) 7535
    3) 1) "10.24.108.1"
       2) (integer) 6379
       3) "9da853291de8f09bb932e8e09a6b48d810a1c962"
    4) 1) "10.24.65.183"
       2) (integer) 6379
       3) "7496fa3a15564b45d603265fb6c8e0444c782738"
41) 1) (integer) 7536
    2) (integer) 8191
    3) 1) "10.24.67.179"
       2) (integer) 6379
       3) "80128f306c2fa8711ce25b3dfa125152e4310d92"
    4) 1) "10.24.96.147"
       2) (integer) 6379
       3) "152adf2b32817c09fae6cd259bdcab01f9584c35"
42) 1) (integer) 8192
    2) (integer) 8314
    3) 1) "10.24.96.124"
       2) (integer) 6379
       3) "075e9f0a1a53569f6c8e6a4821937e614bd12ee9"
    4) 1) "10.24.72.35"
       2) (integer) 6379
       3) "fba74bd3be58e93ccf328914a57407bc68b677af"
43) 1) (integer) 8315
    2) (integer) 8315
    3) 1) "10.24.75.131"
       2) (integer) 6379
       3) "16919de207c198061ededb37c8590532239828ef"
    4) 1) "10.24.82.36"
       2) (integer) 6379
       3) "df68a94dc2a71853ad591e93c1c4405f30455216"
44) 1) (integer) 8316
    2) (integer) 8463
    3) 1) "10.24.96.124"
       2) (integer) 6379
       3) "075e9f0a1a53569f6c8e6a4821937e614bd12ee9"
    4) 1) "10.24.72.35"
       2) (integer) 6379
       3) "fba74bd3be58e93ccf328914a57407bc68b677af"
45) 1) (integer) 8464
    2) (integer) 8464
    3) 1) "10.24.90.220"
       2) (integer) 6379
       3) "b95f503223b92cfb109e4561399339b5702f1beb"
    4) 1) "10.24.107.148"
       2) (integer) 6379
       3) "e81222987fa225e444e64306ad33d30adfffb5ec"
46) 1) (integer) 8465
    2) (integer) 8737
    3) 1) "10.24.70.114"
       2) (integer) 6379
       3) "96c477de9f8f766c8d5ad8e1870df7adac2322e3"
    4) 1) "10.24.82.77"
       2) (integer) 6379
       3) "bcb18281cbdac6c1c514cc2d2a89cd2a27efba8a"
47) 1) (integer) 8738
    2) (integer) 8901
    3) 1) "10.24.69.144"
       2) (integer) 6379
       3) "7924491090d15e1d9405b4e84fa16b5bdd62106f"
    4) 1) "10.24.80.142"
       2) (integer) 6379
       3) "0c58d020fef0f4a416094838c0c54ee59ee83eb4"
48) 1) (integer) 8902
    2) (integer) 9556
    3) 1) "10.24.73.189"
       2) (integer) 6379
       3) "565734a5485c850517eda56d7076ec2b3a6a9d65"
    4) 1) "10.24.80.229"
       2) (integer) 6379
       3) "c8ffe91c3a493ec29d546923e0eab25bd59622f8"
49) 1) (integer) 9557
    2) (integer) 9830
    3) 1) "10.24.88.141"
       2) (integer) 6379
       3) "1bcebfbdcb02cbf33f219c9f2b1b9a072a9b401c"
    4) 1) "10.24.98.0"
       2) (integer) 6379
       3) "c186b182cdbbc306f97ea786f110f9fd92110568"
50) 1) (integer) 9831
    2) (integer) 10103
    3) 1) "10.24.72.95"
       2) (integer) 6379
       3) "5d3c84ccbb22a6b0457876b79ce118e0e527e6b3"
    4) 1) "10.24.85.111"
       2) (integer) 6379
       3) "96914b07338c13f77a3828cdc355398bd164db10"
51) 1) (integer) 10104
    2) (integer) 10267
    3) 1) "10.24.69.144"
       2) (integer) 6379
       3) "7924491090d15e1d9405b4e84fa16b5bdd62106f"
    4) 1) "10.24.80.142"
       2) (integer) 6379
       3) "0c58d020fef0f4a416094838c0c54ee59ee83eb4"
52) 1) (integer) 10268
    2) (integer) 10922
    3) 1) "10.24.81.202"
       2) (integer) 6379
       3) "cb0e71df63c9081af3245c86ebee537e2e8237e4"
    4) 1) "10.24.105.11"
       2) (integer) 6379
       3) "dc9e0b400639917294259b5ac25cc469656b7797"
53) 1) (integer) 10923
    2) (integer) 11195
    3) 1) "10.24.75.131"
       2) (integer) 6379
       3) "16919de207c198061ededb37c8590532239828ef"
    4) 1) "10.24.82.36"
       2) (integer) 6379
       3) "df68a94dc2a71853ad591e93c1c4405f30455216"
54) 1) (integer) 11196
    2) (integer) 11467
    3) 1) "10.24.106.131"
       2) (integer) 6379
       3) "c450835ccc66843c44d9768f390d860fc9633a6b"
    4) 1) "10.24.70.53"
       2) (integer) 6379
       3) "f635a537362533d04e51d0d9dc3d3af840ca1280"
55) 1) (integer) 11468
    2) (integer) 11631
    3) 1) "10.24.106.170"
       2) (integer) 6379
       3) "87b0b8ad2a7eafc41d80163b29658efd321ff089"
    4) 1) "10.24.72.90"
       2) (integer) 6379
       3) "4e1a822779e4b2b68cb8d4584899d477806c5339"
56) 1) (integer) 11632
    2) (integer) 12287
    3) 1) "10.24.102.0"
       2) (integer) 6379
       3) "c28db2886b5e05c9b36b28da03ac54ffc2e3dee3"
    4) 1) "10.24.79.174"
       2) (integer) 6379
       3) "2776b5be74a79da05f872ab88987ec3672c5f004"
57) 1) (integer) 12288
    2) (integer) 12560
    3) 1) "10.24.75.131"
       2) (integer) 6379
       3) "16919de207c198061ededb37c8590532239828ef"
    4) 1) "10.24.82.36"
       2) (integer) 6379
       3) "df68a94dc2a71853ad591e93c1c4405f30455216"
58) 1) (integer) 12561
    2) (integer) 12833
    3) 1) "10.24.70.114"
       2) (integer) 6379
       3) "96c477de9f8f766c8d5ad8e1870df7adac2322e3"
    4) 1) "10.24.82.77"
       2) (integer) 6379
       3) "bcb18281cbdac6c1c514cc2d2a89cd2a27efba8a"
59) 1) (integer) 12834
    2) (integer) 12996
    3) 1) "10.24.108.1"
       2) (integer) 6379
       3) "9da853291de8f09bb932e8e09a6b48d810a1c962"
    4) 1) "10.24.65.183"
       2) (integer) 6379
       3) "7496fa3a15564b45d603265fb6c8e0444c782738"
60) 1) (integer) 12997
    2) (integer) 13652
    3) 1) "10.24.67.254"
       2) (integer) 6379
       3) "29728be273e78753f3b7ad2fefad98b42f6fcec3"
    4) 1) "10.24.91.225"
       2) (integer) 6379
       3) "3c3afe5f7c89560c12c59ab763a8ea47107b9fa6"
61) 1) (integer) 13653
    2) (integer) 13926
    3) 1) "10.24.88.141"
       2) (integer) 6379
       3) "1bcebfbdcb02cbf33f219c9f2b1b9a072a9b401c"
    4) 1) "10.24.98.0"
       2) (integer) 6379
       3) "c186b182cdbbc306f97ea786f110f9fd92110568"
62) 1) (integer) 13927
    2) (integer) 14199
    3) 1) "10.24.82.226"
       2) (integer) 6379
       3) "d8601f4f05eb728c718cb2c696a6e0dd1d861f64"
    4) 1) "10.24.96.34"
       2) (integer) 6379
       3) "6ca0e6e793c2d73601e566017b3920ce0a032c99"
63) 1) (integer) 14200
    2) (integer) 14200
    3) 1) "10.24.79.25"
       2) (integer) 6379
       3) "df17ee6c3aa5aa4d6cd84cbf640737d30d265dc0"
    4) 1) "10.24.82.90"
       2) (integer) 6379
       3) "d34559353ce7d57f3d7f09041e414f659b32b480"
64) 1) (integer) 14201
    2) (integer) 14363
    3) 1) "10.24.87.22"
       2) (integer) 6379
       3) "0be8e0a01cd11c770140e5e1aacbc7e11f2047b9"
    4) 1) "10.24.109.70"
       2) (integer) 6379
       3) "f5a9cfd1a66cc797a32b583efb3da782b45da02a"
65) 1) (integer) 14364
    2) (integer) 15018
    3) 1) "10.24.92.75"
       2) (integer) 6379
       3) "de3e35b1b2e91d1ed503bbfa26bcb20608989453"
    4) 1) "10.24.98.235"
       2) (integer) 6379
       3) "64ab49d38b0a7bf691c28a62d4cf1258b8c4b73d"
66) 1) (integer) 15019
    2) (integer) 15291
    3) 1) "10.24.96.124"
       2) (integer) 6379
       3) "075e9f0a1a53569f6c8e6a4821937e614bd12ee9"
    4) 1) "10.24.72.35"
       2) (integer) 6379
       3) "fba74bd3be58e93ccf328914a57407bc68b677af"
67) 1) (integer) 15292
    2) (integer) 15563
    3) 1) "10.24.106.131"
       2) (integer) 6379
       3) "c450835ccc66843c44d9768f390d860fc9633a6b"
    4) 1) "10.24.70.53"
       2) (integer) 6379
       3) "f635a537362533d04e51d0d9dc3d3af840ca1280"
68) 1) (integer) 15564
    2) (integer) 15727
    3) 1) "10.24.106.170"
       2) (integer) 6379
       3) "87b0b8ad2a7eafc41d80163b29658efd321ff089"
    4) 1) "10.24.72.90"
       2) (integer) 6379
       3) "4e1a822779e4b2b68cb8d4584899d477806c5339"
69) 1) (integer) 15728
    2) (integer) 16383
    3) 1) "10.24.110.223"
       2) (integer) 6379
       3) "3720b2c0cded538c9fdbad851a40bd02b58358ed"
    4) 1) "10.24.73.3"
       2) (integer) 6379
       3) "587523f1d707f9186941f7e1e7f9ed39345d0612"

no issue version (one-line reverted)

root@tim-zhu-shell:/# redis-cli -h redis-xxxx.net -p 6379
redis-xxxx.net:6379> CLUSTER NODES
df68a94dc2a71853ad591e93c1c4405f30455216 10.24.82.36:6379@16379 slave 16919de207c198061ededb37c8590532239828ef 0 1741704287000 20 connected
7924491090d15e1d9405b4e84fa16b5bdd62106f 10.24.69.144:6379@16379 master - 0 1741704287000 53 connected 1370-1530 1640-1641 6828-6991 8738-8901 10104-10267
bcb18281cbdac6c1c514cc2d2a89cd2a27efba8a 10.24.82.77:6379@16379 slave 96c477de9f8f766c8d5ad8e1870df7adac2322e3 0 1741704286000 36 connected
6ca0e6e793c2d73601e566017b3920ce0a032c99 10.24.96.34:6379@16379 slave d8601f4f05eb728c718cb2c696a6e0dd1d861f64 0 1741704287000 38 connected
fba74bd3be58e93ccf328914a57407bc68b677af 10.24.72.35:6379@16379 slave 075e9f0a1a53569f6c8e6a4821937e614bd12ee9 0 1741704287000 21 connected
cb0e71df63c9081af3245c86ebee537e2e8237e4 10.24.81.202:6379@16379 master - 0 1741704287000 8 connected 10268-10922
9da853291de8f09bb932e8e09a6b48d810a1c962 10.24.108.1:6379@16379 myself,master - 0 1741704286000 52 connected 1367-1369 4369-4532 4642-4804 7373-7535 12834-12996
1c72a09c55fef2a5a05bd40fac7ffeb8aeb5b995 10.24.89.51:6379@16379 master - 0 1741704286000 5 connected 6172-6826
152adf2b32817c09fae6cd259bdcab01f9584c35 10.24.96.147:6379@16379 slave 80128f306c2fa8711ce25b3dfa125152e4310d92 0 1741704288558 42 connected
4b4a74b1ed6ef93a84ddda7c4f3a609335ccf125 10.24.108.244:6379@16379 slave 1c72a09c55fef2a5a05bd40fac7ffeb8aeb5b995 0 1741704287000 5 connected
0be8e0a01cd11c770140e5e1aacbc7e11f2047b9 10.24.87.22:6379@16379 master - 0 1741704287000 55 connected 0-163 547-710 1637-1638 4096-4257 14201-14363
3c3afe5f7c89560c12c59ab763a8ea47107b9fa6 10.24.91.225:6379@16379 slave 29728be273e78753f3b7ad2fefad98b42f6fcec3 0 1741704288000 10 connected
075e9f0a1a53569f6c8e6a4821937e614bd12ee9 10.24.96.124:6379@16379 master - 0 1741704288000 21 connected 4258-4368 8192-8314 8316-8463 15019-15291
7496fa3a15564b45d603265fb6c8e0444c782738 10.24.65.183:6379@16379 slave 9da853291de8f09bb932e8e09a6b48d810a1c962 0 1741704287000 52 connected
80128f306c2fa8711ce25b3dfa125152e4310d92 10.24.67.179:6379@16379 master - 0 1741704284000 42 connected 7536-8191
1bcebfbdcb02cbf33f219c9f2b1b9a072a9b401c 10.24.88.141:6379@16379 master - 0 1741704287000 18 connected 5628-5734 9557-9830 13653-13926
d8601f4f05eb728c718cb2c696a6e0dd1d861f64 10.24.82.226:6379@16379 master - 0 1741704287000 38 connected 1804-1912 5735-6007 13927-14199
e81222987fa225e444e64306ad33d30adfffb5ec 10.24.107.148:6379@16379 slave b95f503223b92cfb109e4561399339b5702f1beb 0 1741704287000 22 connected
c450835ccc66843c44d9768f390d860fc9633a6b 10.24.106.131:6379@16379 master - 0 1741704287000 35 connected 3164-3275 11196-11467 15292-15563
a902832eb3b9cc78c95da2e383c0335058835ed7 10.24.109.201:6379@16379 slave 4855173a8e10c9209eedbe101de8a36aa8692fbb 0 1741704286000 39 connected
d34559353ce7d57f3d7f09041e414f659b32b480 10.24.82.90:6379@16379 slave df17ee6c3aa5aa4d6cd84cbf640737d30d265dc0 0 1741704286000 54 connected
df17ee6c3aa5aa4d6cd84cbf640737d30d265dc0 10.24.79.25:6379@16379 master - 0 1741704286000 54 connected 1642-1803 1913-2076 5464-5627 6008-6171 14200
96914b07338c13f77a3828cdc355398bd164db10 10.24.85.111:6379@16379 slave 5d3c84ccbb22a6b0457876b79ce118e0e527e6b3 0 1741704286000 37 connected
2776b5be74a79da05f872ab88987ec3672c5f004 10.24.79.174:6379@16379 slave c28db2886b5e05c9b36b28da03ac54ffc2e3dee3 0 1741704286000 27 connected
16919de207c198061ededb37c8590532239828ef 10.24.75.131:6379@16379 master - 0 1741704287000 20 connected 6992-7099 8315 10923-11195 12288-12560
4e1a822779e4b2b68cb8d4584899d477806c5339 10.24.72.90:6379@16379 slave 87b0b8ad2a7eafc41d80163b29658efd321ff089 0 1741704287000 51 connected
f5a9cfd1a66cc797a32b583efb3da782b45da02a 10.24.109.70:6379@16379 slave 0be8e0a01cd11c770140e5e1aacbc7e11f2047b9 0 1741704286000 55 connected
4855173a8e10c9209eedbe101de8a36aa8692fbb 10.24.94.166:6379@16379 master - 0 1741704287000 39 connected 164-272 274-546 7100-7372
5d3c84ccbb22a6b0457876b79ce118e0e527e6b3 10.24.72.95:6379@16379 master - 0 1741704286000 37 connected 1531-1636 2731-3002 5461-5463 6827 9831-10103
f635a537362533d04e51d0d9dc3d3af840ca1280 10.24.70.53:6379@16379 slave c450835ccc66843c44d9768f390d860fc9633a6b 0 1741704286000 35 connected
96c477de9f8f766c8d5ad8e1870df7adac2322e3 10.24.70.114:6379@16379 master - 0 1741704288558 36 connected 4533-4641 8465-8737 12561-12833
587523f1d707f9186941f7e1e7f9ed39345d0612 10.24.73.3:6379@16379 slave 3720b2c0cded538c9fdbad851a40bd02b58358ed 0 1741704284000 23 connected
3eaf634b52441c9e6a31398e50e55e7c1112ae84 10.24.95.63:6379@16379 master - 0 1741704284000 25 connected 711-1364 3003
9524fc5ffbc0e44651c0f96fe8f8f842fdc10bd0 10.24.100.82:6379@16379 master - 0 1741704286000 26 connected 3440-4095
ffead3ee63fab8831959124a5eee93da75397b7a 10.24.92.116:6379@16379 slave c92b87eec32eaf7da0aa73fdf5694c3c5d23f59d 0 1741704287000 24 connected
c28db2886b5e05c9b36b28da03ac54ffc2e3dee3 10.24.102.0:6379@16379 master - 0 1741704287000 27 connected 11632-12287
c186b182cdbbc306f97ea786f110f9fd92110568 10.24.98.0:6379@16379 slave 1bcebfbdcb02cbf33f219c9f2b1b9a072a9b401c 0 1741704287000 18 connected
565734a5485c850517eda56d7076ec2b3a6a9d65 10.24.73.189:6379@16379 master - 0 1741704287000 7 connected 8902-9556
873a694cdb740c71e988a6240f518eec00080f36 10.24.66.51:6379@16379 slave 3eaf634b52441c9e6a31398e50e55e7c1112ae84 0 1741704284000 25 connected
3720b2c0cded538c9fdbad851a40bd02b58358ed 10.24.110.223:6379@16379 master - 0 1741704286000 23 connected 15728-16383
c8ffe91c3a493ec29d546923e0eab25bd59622f8 10.24.80.229:6379@16379 slave 565734a5485c850517eda56d7076ec2b3a6a9d65 0 1741704287000 7 connected
29728be273e78753f3b7ad2fefad98b42f6fcec3 10.24.67.254:6379@16379 master - 0 1741704286000 10 connected 12997-13652
64ab49d38b0a7bf691c28a62d4cf1258b8c4b73d 10.24.98.235:6379@16379 slave de3e35b1b2e91d1ed503bbfa26bcb20608989453 0 1741704284000 11 connected
c92b87eec32eaf7da0aa73fdf5694c3c5d23f59d 10.24.66.212:6379@16379 master - 0 1741704285000 24 connected 4805-5460
dc9e0b400639917294259b5ac25cc469656b7797 10.24.105.11:6379@16379 slave cb0e71df63c9081af3245c86ebee537e2e8237e4 0 1741704287000 8 connected
87b0b8ad2a7eafc41d80163b29658efd321ff089 10.24.106.170:6379@16379 master - 0 1741704287000 51 connected 273 1365-1366 1639 3004-3163 3276-3439 11468-11631 15564-15727
de3e35b1b2e91d1ed503bbfa26bcb20608989453 10.24.92.75:6379@16379 master - 0 1741704286000 11 connected 14364-15018
d5248d7b96f941b8931544aebdfc6cfcfdd289d9 10.24.70.32:6379@16379 slave 9524fc5ffbc0e44651c0f96fe8f8f842fdc10bd0 0 1741704284000 26 connected
0c58d020fef0f4a416094838c0c54ee59ee83eb4 10.24.80.142:6379@16379 slave 7924491090d15e1d9405b4e84fa16b5bdd62106f 0 1741704287000 53 connected
b95f503223b92cfb109e4561399339b5702f1beb 10.24.90.220:6379@16379 master - 0 1741704285000 22 connected 2077-2730 8464

redis-xxxx.net:6379> CLUSTER SLOTS
1) 1) (integer) 0
  2) (integer) 163
  3) 1) "10.24.87.22"
     2) (integer) 6379
     3) "0be8e0a01cd11c770140e5e1aacbc7e11f2047b9"
  4) 1) "10.24.109.70"
     2) (integer) 6379
     3) "f5a9cfd1a66cc797a32b583efb3da782b45da02a"
2) 1) (integer) 164
  2) (integer) 272
  3) 1) "10.24.94.166"
     2) (integer) 6379
     3) "4855173a8e10c9209eedbe101de8a36aa8692fbb"
  4) 1) "10.24.109.201"
     2) (integer) 6379
     3) "a902832eb3b9cc78c95da2e383c0335058835ed7"
3) 1) (integer) 273
  2) (integer) 273
  3) 1) "10.24.106.170"
     2) (integer) 6379
     3) "87b0b8ad2a7eafc41d80163b29658efd321ff089"
  4) 1) "10.24.72.90"
     2) (integer) 6379
     3) "4e1a822779e4b2b68cb8d4584899d477806c5339"
4) 1) (integer) 274
  2) (integer) 546
  3) 1) "10.24.94.166"
     2) (integer) 6379
     3) "4855173a8e10c9209eedbe101de8a36aa8692fbb"
  4) 1) "10.24.109.201"
     2) (integer) 6379
     3) "a902832eb3b9cc78c95da2e383c0335058835ed7"
5) 1) (integer) 547
  2) (integer) 710
  3) 1) "10.24.87.22"
     2) (integer) 6379
     3) "0be8e0a01cd11c770140e5e1aacbc7e11f2047b9"
  4) 1) "10.24.109.70"
     2) (integer) 6379
     3) "f5a9cfd1a66cc797a32b583efb3da782b45da02a"
6) 1) (integer) 711
  2) (integer) 1364
  3) 1) "10.24.95.63"
     2) (integer) 6379
     3) "3eaf634b52441c9e6a31398e50e55e7c1112ae84"
  4) 1) "10.24.66.51"
     2) (integer) 6379
     3) "873a694cdb740c71e988a6240f518eec00080f36"
7) 1) (integer) 1365
  2) (integer) 1366
  3) 1) "10.24.106.170"
     2) (integer) 6379
     3) "87b0b8ad2a7eafc41d80163b29658efd321ff089"
  4) 1) "10.24.72.90"
     2) (integer) 6379
     3) "4e1a822779e4b2b68cb8d4584899d477806c5339"
8) 1) (integer) 1367
  2) (integer) 1369
  3) 1) "10.24.108.1"
     2) (integer) 6379
     3) "9da853291de8f09bb932e8e09a6b48d810a1c962"
  4) 1) "10.24.65.183"
     2) (integer) 6379
     3) "7496fa3a15564b45d603265fb6c8e0444c782738"
9) 1) (integer) 1370
  2) (integer) 1530
  3) 1) "10.24.69.144"
     2) (integer) 6379
     3) "7924491090d15e1d9405b4e84fa16b5bdd62106f"
  4) 1) "10.24.80.142"
     2) (integer) 6379
     3) "0c58d020fef0f4a416094838c0c54ee59ee83eb4"
10) 1) (integer) 1531
  2) (integer) 1636
  3) 1) "10.24.72.95"
     2) (integer) 6379
     3) "5d3c84ccbb22a6b0457876b79ce118e0e527e6b3"
  4) 1) "10.24.85.111"
     2) (integer) 6379
     3) "96914b07338c13f77a3828cdc355398bd164db10"
11) 1) (integer) 1637
  2) (integer) 1638
  3) 1) "10.24.87.22"
     2) (integer) 6379
     3) "0be8e0a01cd11c770140e5e1aacbc7e11f2047b9"
  4) 1) "10.24.109.70"
     2) (integer) 6379
     3) "f5a9cfd1a66cc797a32b583efb3da782b45da02a"
12) 1) (integer) 1639
  2) (integer) 1639
  3) 1) "10.24.106.170"
     2) (integer) 6379
     3) "87b0b8ad2a7eafc41d80163b29658efd321ff089"
  4) 1) "10.24.72.90"
     2) (integer) 6379
     3) "4e1a822779e4b2b68cb8d4584899d477806c5339"
13) 1) (integer) 1640
  2) (integer) 1641
  3) 1) "10.24.69.144"
     2) (integer) 6379
     3) "7924491090d15e1d9405b4e84fa16b5bdd62106f"
  4) 1) "10.24.80.142"
     2) (integer) 6379
     3) "0c58d020fef0f4a416094838c0c54ee59ee83eb4"
14) 1) (integer) 1642
  2) (integer) 1803
  3) 1) "10.24.79.25"
     2) (integer) 6379
     3) "df17ee6c3aa5aa4d6cd84cbf640737d30d265dc0"
  4) 1) "10.24.82.90"
     2) (integer) 6379
     3) "d34559353ce7d57f3d7f09041e414f659b32b480"
15) 1) (integer) 1804
  2) (integer) 1912
  3) 1) "10.24.82.226"
     2) (integer) 6379
     3) "d8601f4f05eb728c718cb2c696a6e0dd1d861f64"
  4) 1) "10.24.96.34"
     2) (integer) 6379
     3) "6ca0e6e793c2d73601e566017b3920ce0a032c99"
16) 1) (integer) 1913
  2) (integer) 2076
  3) 1) "10.24.79.25"
     2) (integer) 6379
     3) "df17ee6c3aa5aa4d6cd84cbf640737d30d265dc0"
  4) 1) "10.24.82.90"
     2) (integer) 6379
     3) "d34559353ce7d57f3d7f09041e414f659b32b480"
17) 1) (integer) 2077
  2) (integer) 2730
  3) 1) "10.24.90.220"
     2) (integer) 6379
     3) "b95f503223b92cfb109e4561399339b5702f1beb"
  4) 1) "10.24.107.148"
     2) (integer) 6379
     3) "e81222987fa225e444e64306ad33d30adfffb5ec"
18) 1) (integer) 2731
  2) (integer) 3002
  3) 1) "10.24.72.95"
     2) (integer) 6379
     3) "5d3c84ccbb22a6b0457876b79ce118e0e527e6b3"
  4) 1) "10.24.85.111"
     2) (integer) 6379
     3) "96914b07338c13f77a3828cdc355398bd164db10"
19) 1) (integer) 3003
  2) (integer) 3003
  3) 1) "10.24.95.63"
     2) (integer) 6379
     3) "3eaf634b52441c9e6a31398e50e55e7c1112ae84"
  4) 1) "10.24.66.51"
     2) (integer) 6379
     3) "873a694cdb740c71e988a6240f518eec00080f36"
20) 1) (integer) 3004
  2) (integer) 3163
  3) 1) "10.24.106.170"
     2) (integer) 6379
     3) "87b0b8ad2a7eafc41d80163b29658efd321ff089"
  4) 1) "10.24.72.90"
     2) (integer) 6379
     3) "4e1a822779e4b2b68cb8d4584899d477806c5339"
21) 1) (integer) 3164
  2) (integer) 3275
  3) 1) "10.24.106.131"
     2) (integer) 6379
     3) "c450835ccc66843c44d9768f390d860fc9633a6b"
  4) 1) "10.24.70.53"
     2) (integer) 6379
     3) "f635a537362533d04e51d0d9dc3d3af840ca1280"
22) 1) (integer) 3276
  2) (integer) 3439
  3) 1) "10.24.106.170"
     2) (integer) 6379
     3) "87b0b8ad2a7eafc41d80163b29658efd321ff089"
  4) 1) "10.24.72.90"
     2) (integer) 6379
     3) "4e1a822779e4b2b68cb8d4584899d477806c5339"
23) 1) (integer) 3440
  2) (integer) 4095
  3) 1) "10.24.100.82"
     2) (integer) 6379
     3) "9524fc5ffbc0e44651c0f96fe8f8f842fdc10bd0"
  4) 1) "10.24.70.32"
     2) (integer) 6379
     3) "d5248d7b96f941b8931544aebdfc6cfcfdd289d9"
24) 1) (integer) 4096
  2) (integer) 4257
  3) 1) "10.24.87.22"
     2) (integer) 6379
     3) "0be8e0a01cd11c770140e5e1aacbc7e11f2047b9"
  4) 1) "10.24.109.70"
     2) (integer) 6379
     3) "f5a9cfd1a66cc797a32b583efb3da782b45da02a"
25) 1) (integer) 4258
  2) (integer) 4368
  3) 1) "10.24.96.124"
     2) (integer) 6379
     3) "075e9f0a1a53569f6c8e6a4821937e614bd12ee9"
  4) 1) "10.24.72.35"
     2) (integer) 6379
     3) "fba74bd3be58e93ccf328914a57407bc68b677af"
26) 1) (integer) 4369
  2) (integer) 4532
  3) 1) "10.24.108.1"
     2) (integer) 6379
     3) "9da853291de8f09bb932e8e09a6b48d810a1c962"
  4) 1) "10.24.65.183"
     2) (integer) 6379
     3) "7496fa3a15564b45d603265fb6c8e0444c782738"
27) 1) (integer) 4533
  2) (integer) 4641
  3) 1) "10.24.70.114"
     2) (integer) 6379
     3) "96c477de9f8f766c8d5ad8e1870df7adac2322e3"
  4) 1) "10.24.82.77"
     2) (integer) 6379
     3) "bcb18281cbdac6c1c514cc2d2a89cd2a27efba8a"
28) 1) (integer) 4642
  2) (integer) 4804
  3) 1) "10.24.108.1"
     2) (integer) 6379
     3) "9da853291de8f09bb932e8e09a6b48d810a1c962"
  4) 1) "10.24.65.183"
     2) (integer) 6379
     3) "7496fa3a15564b45d603265fb6c8e0444c782738"
29) 1) (integer) 4805
  2) (integer) 5460
  3) 1) "10.24.66.212"
     2) (integer) 6379
     3) "c92b87eec32eaf7da0aa73fdf5694c3c5d23f59d"
  4) 1) "10.24.92.116"
     2) (integer) 6379
     3) "ffead3ee63fab8831959124a5eee93da75397b7a"
30) 1) (integer) 5461
  2) (integer) 5463
  3) 1) "10.24.72.95"
     2) (integer) 6379
     3) "5d3c84ccbb22a6b0457876b79ce118e0e527e6b3"
  4) 1) "10.24.85.111"
     2) (integer) 6379
     3) "96914b07338c13f77a3828cdc355398bd164db10"
31) 1) (integer) 5464
  2) (integer) 5627
  3) 1) "10.24.79.25"
     2) (integer) 6379
     3) "df17ee6c3aa5aa4d6cd84cbf640737d30d265dc0"
  4) 1) "10.24.82.90"
     2) (integer) 6379
     3) "d34559353ce7d57f3d7f09041e414f659b32b480"
32) 1) (integer) 5628
  2) (integer) 5734
  3) 1) "10.24.88.141"
     2) (integer) 6379
     3) "1bcebfbdcb02cbf33f219c9f2b1b9a072a9b401c"
  4) 1) "10.24.98.0"
     2) (integer) 6379
     3) "c186b182cdbbc306f97ea786f110f9fd92110568"
33) 1) (integer) 5735
  2) (integer) 6007
  3) 1) "10.24.82.226"
     2) (integer) 6379
     3) "d8601f4f05eb728c718cb2c696a6e0dd1d861f64"
  4) 1) "10.24.96.34"
     2) (integer) 6379
     3) "6ca0e6e793c2d73601e566017b3920ce0a032c99"
34) 1) (integer) 6008
  2) (integer) 6171
  3) 1) "10.24.79.25"
     2) (integer) 6379
     3) "df17ee6c3aa5aa4d6cd84cbf640737d30d265dc0"
  4) 1) "10.24.82.90"
     2) (integer) 6379
     3) "d34559353ce7d57f3d7f09041e414f659b32b480"
35) 1) (integer) 6172
  2) (integer) 6826
  3) 1) "10.24.89.51"
     2) (integer) 6379
     3) "1c72a09c55fef2a5a05bd40fac7ffeb8aeb5b995"
  4) 1) "10.24.108.244"
     2) (integer) 6379
     3) "4b4a74b1ed6ef93a84ddda7c4f3a609335ccf125"
36) 1) (integer) 6827
  2) (integer) 6827
  3) 1) "10.24.72.95"
     2) (integer) 6379
     3) "5d3c84ccbb22a6b0457876b79ce118e0e527e6b3"
  4) 1) "10.24.85.111"
     2) (integer) 6379
     3) "96914b07338c13f77a3828cdc355398bd164db10"
37) 1) (integer) 6828
  2) (integer) 6991
  3) 1) "10.24.69.144"
     2) (integer) 6379
     3) "7924491090d15e1d9405b4e84fa16b5bdd62106f"
  4) 1) "10.24.80.142"
     2) (integer) 6379
     3) "0c58d020fef0f4a416094838c0c54ee59ee83eb4"
38) 1) (integer) 6992
  2) (integer) 7099
  3) 1) "10.24.75.131"
     2) (integer) 6379
     3) "16919de207c198061ededb37c8590532239828ef"
  4) 1) "10.24.82.36"
     2) (integer) 6379
     3) "df68a94dc2a71853ad591e93c1c4405f30455216"
39) 1) (integer) 7100
  2) (integer) 7372
  3) 1) "10.24.94.166"
     2) (integer) 6379
     3) "4855173a8e10c9209eedbe101de8a36aa8692fbb"
  4) 1) "10.24.109.201"
     2) (integer) 6379
     3) "a902832eb3b9cc78c95da2e383c0335058835ed7"
40) 1) (integer) 7373
  2) (integer) 7535
  3) 1) "10.24.108.1"
     2) (integer) 6379
     3) "9da853291de8f09bb932e8e09a6b48d810a1c962"
  4) 1) "10.24.65.183"
     2) (integer) 6379
     3) "7496fa3a15564b45d603265fb6c8e0444c782738"
41) 1) (integer) 7536
  2) (integer) 8191
  3) 1) "10.24.67.179"
     2) (integer) 6379
     3) "80128f306c2fa8711ce25b3dfa125152e4310d92"
  4) 1) "10.24.96.147"
     2) (integer) 6379
     3) "152adf2b32817c09fae6cd259bdcab01f9584c35"
42) 1) (integer) 8192
  2) (integer) 8314
  3) 1) "10.24.96.124"
     2) (integer) 6379
     3) "075e9f0a1a53569f6c8e6a4821937e614bd12ee9"
  4) 1) "10.24.72.35"
     2) (integer) 6379
     3) "fba74bd3be58e93ccf328914a57407bc68b677af"
43) 1) (integer) 8315
  2) (integer) 8315
  3) 1) "10.24.75.131"
     2) (integer) 6379
     3) "16919de207c198061ededb37c8590532239828ef"
  4) 1) "10.24.82.36"
     2) (integer) 6379
     3) "df68a94dc2a71853ad591e93c1c4405f30455216"
44) 1) (integer) 8316
  2) (integer) 8463
  3) 1) "10.24.96.124"
     2) (integer) 6379
     3) "075e9f0a1a53569f6c8e6a4821937e614bd12ee9"
  4) 1) "10.24.72.35"
     2) (integer) 6379
     3) "fba74bd3be58e93ccf328914a57407bc68b677af"
45) 1) (integer) 8464
  2) (integer) 8464
  3) 1) "10.24.90.220"
     2) (integer) 6379
     3) "b95f503223b92cfb109e4561399339b5702f1beb"
  4) 1) "10.24.107.148"
     2) (integer) 6379
     3) "e81222987fa225e444e64306ad33d30adfffb5ec"
46) 1) (integer) 8465
  2) (integer) 8737
  3) 1) "10.24.70.114"
     2) (integer) 6379
     3) "96c477de9f8f766c8d5ad8e1870df7adac2322e3"
  4) 1) "10.24.82.77"
     2) (integer) 6379
     3) "bcb18281cbdac6c1c514cc2d2a89cd2a27efba8a"
47) 1) (integer) 8738
  2) (integer) 8901
  3) 1) "10.24.69.144"
     2) (integer) 6379
     3) "7924491090d15e1d9405b4e84fa16b5bdd62106f"
  4) 1) "10.24.80.142"
     2) (integer) 6379
     3) "0c58d020fef0f4a416094838c0c54ee59ee83eb4"
48) 1) (integer) 8902
  2) (integer) 9556
  3) 1) "10.24.73.189"
     2) (integer) 6379
     3) "565734a5485c850517eda56d7076ec2b3a6a9d65"
  4) 1) "10.24.80.229"
     2) (integer) 6379
     3) "c8ffe91c3a493ec29d546923e0eab25bd59622f8"
49) 1) (integer) 9557
  2) (integer) 9830
  3) 1) "10.24.88.141"
     2) (integer) 6379
     3) "1bcebfbdcb02cbf33f219c9f2b1b9a072a9b401c"
  4) 1) "10.24.98.0"
     2) (integer) 6379
     3) "c186b182cdbbc306f97ea786f110f9fd92110568"
50) 1) (integer) 9831
  2) (integer) 10103
  3) 1) "10.24.72.95"
     2) (integer) 6379
     3) "5d3c84ccbb22a6b0457876b79ce118e0e527e6b3"
  4) 1) "10.24.85.111"
     2) (integer) 6379
     3) "96914b07338c13f77a3828cdc355398bd164db10"
51) 1) (integer) 10104
  2) (integer) 10267
  3) 1) "10.24.69.144"
     2) (integer) 6379
     3) "7924491090d15e1d9405b4e84fa16b5bdd62106f"
  4) 1) "10.24.80.142"
     2) (integer) 6379
     3) "0c58d020fef0f4a416094838c0c54ee59ee83eb4"
52) 1) (integer) 10268
  2) (integer) 10922
  3) 1) "10.24.81.202"
     2) (integer) 6379
     3) "cb0e71df63c9081af3245c86ebee537e2e8237e4"
  4) 1) "10.24.105.11"
     2) (integer) 6379
     3) "dc9e0b400639917294259b5ac25cc469656b7797"
53) 1) (integer) 10923
  2) (integer) 11195
  3) 1) "10.24.75.131"
     2) (integer) 6379
     3) "16919de207c198061ededb37c8590532239828ef"
  4) 1) "10.24.82.36"
     2) (integer) 6379
     3) "df68a94dc2a71853ad591e93c1c4405f30455216"
54) 1) (integer) 11196
  2) (integer) 11467
  3) 1) "10.24.106.131"
     2) (integer) 6379
     3) "c450835ccc66843c44d9768f390d860fc9633a6b"
  4) 1) "10.24.70.53"
     2) (integer) 6379
     3) "f635a537362533d04e51d0d9dc3d3af840ca1280"
55) 1) (integer) 11468
  2) (integer) 11631
  3) 1) "10.24.106.170"
     2) (integer) 6379
     3) "87b0b8ad2a7eafc41d80163b29658efd321ff089"
  4) 1) "10.24.72.90"
     2) (integer) 6379
     3) "4e1a822779e4b2b68cb8d4584899d477806c5339"
56) 1) (integer) 11632
  2) (integer) 12287
  3) 1) "10.24.102.0"
     2) (integer) 6379
     3) "c28db2886b5e05c9b36b28da03ac54ffc2e3dee3"
  4) 1) "10.24.79.174"
     2) (integer) 6379
     3) "2776b5be74a79da05f872ab88987ec3672c5f004"
57) 1) (integer) 12288
  2) (integer) 12560
  3) 1) "10.24.75.131"
     2) (integer) 6379
     3) "16919de207c198061ededb37c8590532239828ef"
  4) 1) "10.24.82.36"
     2) (integer) 6379
     3) "df68a94dc2a71853ad591e93c1c4405f30455216"
58) 1) (integer) 12561
  2) (integer) 12833
  3) 1) "10.24.70.114"
     2) (integer) 6379
     3) "96c477de9f8f766c8d5ad8e1870df7adac2322e3"
  4) 1) "10.24.82.77"
     2) (integer) 6379
     3) "bcb18281cbdac6c1c514cc2d2a89cd2a27efba8a"
59) 1) (integer) 12834
  2) (integer) 12996
  3) 1) "10.24.108.1"
     2) (integer) 6379
     3) "9da853291de8f09bb932e8e09a6b48d810a1c962"
  4) 1) "10.24.65.183"
     2) (integer) 6379
     3) "7496fa3a15564b45d603265fb6c8e0444c782738"
60) 1) (integer) 12997
  2) (integer) 13652
  3) 1) "10.24.67.254"
     2) (integer) 6379
     3) "29728be273e78753f3b7ad2fefad98b42f6fcec3"
  4) 1) "10.24.91.225"
     2) (integer) 6379
     3) "3c3afe5f7c89560c12c59ab763a8ea47107b9fa6"
61) 1) (integer) 13653
  2) (integer) 13926
  3) 1) "10.24.88.141"
     2) (integer) 6379
     3) "1bcebfbdcb02cbf33f219c9f2b1b9a072a9b401c"
  4) 1) "10.24.98.0"
     2) (integer) 6379
     3) "c186b182cdbbc306f97ea786f110f9fd92110568"
62) 1) (integer) 13927
  2) (integer) 14199
  3) 1) "10.24.82.226"
     2) (integer) 6379
     3) "d8601f4f05eb728c718cb2c696a6e0dd1d861f64"
  4) 1) "10.24.96.34"
     2) (integer) 6379
     3) "6ca0e6e793c2d73601e566017b3920ce0a032c99"
63) 1) (integer) 14200
  2) (integer) 14200
  3) 1) "10.24.79.25"
     2) (integer) 6379
     3) "df17ee6c3aa5aa4d6cd84cbf640737d30d265dc0"
  4) 1) "10.24.82.90"
     2) (integer) 6379
     3) "d34559353ce7d57f3d7f09041e414f659b32b480"
64) 1) (integer) 14201
  2) (integer) 14363
  3) 1) "10.24.87.22"
     2) (integer) 6379
     3) "0be8e0a01cd11c770140e5e1aacbc7e11f2047b9"
  4) 1) "10.24.109.70"
     2) (integer) 6379
     3) "f5a9cfd1a66cc797a32b583efb3da782b45da02a"
65) 1) (integer) 14364
  2) (integer) 15018
  3) 1) "10.24.92.75"
     2) (integer) 6379
     3) "de3e35b1b2e91d1ed503bbfa26bcb20608989453"
  4) 1) "10.24.98.235"
     2) (integer) 6379
     3) "64ab49d38b0a7bf691c28a62d4cf1258b8c4b73d"
66) 1) (integer) 15019
  2) (integer) 15291
  3) 1) "10.24.96.124"
     2) (integer) 6379
     3) "075e9f0a1a53569f6c8e6a4821937e614bd12ee9"
  4) 1) "10.24.72.35"
     2) (integer) 6379
     3) "fba74bd3be58e93ccf328914a57407bc68b677af"
67) 1) (integer) 15292
  2) (integer) 15563
  3) 1) "10.24.106.131"
     2) (integer) 6379
     3) "c450835ccc66843c44d9768f390d860fc9633a6b"
  4) 1) "10.24.70.53"
     2) (integer) 6379
     3) "f635a537362533d04e51d0d9dc3d3af840ca1280"
68) 1) (integer) 15564
  2) (integer) 15727
  3) 1) "10.24.106.170"
     2) (integer) 6379
     3) "87b0b8ad2a7eafc41d80163b29658efd321ff089"
  4) 1) "10.24.72.90"
     2) (integer) 6379
     3) "4e1a822779e4b2b68cb8d4584899d477806c5339"
69) 1) (integer) 15728
  2) (integer) 16383
  3) 1) "10.24.110.223"
     2) (integer) 6379
     3) "3720b2c0cded538c9fdbad851a40bd02b58358ed"
  4) 1) "10.24.73.3"
     2) (integer) 6379
     3) "587523f1d707f9186941f7e1e7f9ed39345d0612"
     

[tim-zhu-reddit]

our pipelin is consist of a batch of 25 SET command with NX and TTL

		setArgs := redis.SetArgs{
			Mode: "NX",
			TTL:  client.CacheTTL,
			Get:  false,
		}

[monkey92t]

Based on my understanding, I’m quite puzzled. From your test, it seems that even when you set opt.ReadOnly to false, the issue persists. However, from your comments, it appears that when you reverted that line of code, an error occurred:

if c.opt.ReadOnly && isBadConn(err, false, node.Client.getAddr()) {
    node.MarkAsFailing()
}

There’s an obvious conflict here. According to the code, in the PR, the new addition isBadConn(err, false, node.Client.getAddr()) was introduced, but it’s conditional on opt.ReadOnly being true. Yet in your test, you set opt.ReadOnly to false and still encountered a failure, which suggests that there might be other issues at play—potentially even multiple failures in different places.

You executed a large number of SET commands (write operations) through a pipeline, which should be routed to the master node. go-redis assumes that the first node returned by the CLUSTER SLOTS command is the master node (this is typically how Redis behaves). If possible, could you provide the output of the CLUSTER SLOTS command?

[tim-zhu-reddit]

could you provide the output of the CLUSTER SLOTS command?

@monkey92t I did provide the CLUSTER SLOTS output in #3282 (comment), there was no obvious difference between the readonly=false, latest master broken version and the one-line-reverted working version to me

there might be other issues at play—potentially even multiple failures in different places.

agree, potentially turning readOnly off overwhelmed our redis cluster, and caused timeout errors to happen more, not necessary due to the bug. We had readOnly turned on because it successfully made the cluster more performant and reduced latency and timeouts significantly.

[ndyakov]

So, @tim-zhu-reddit is there a log of the errors that are marking some of the nodes as failing? I will look more into this in the following week. Could it be that you are receiving a lot of moved errors? The timeouts that you have set seem high, so I wouldn't worry about that, but you can see if setting the second argument of isBadConn to true changes the behaviour of the application

[monkey92t]

I reviewed the output results of your cluster nodes and cluster slots commands, and I didn’t find any obvious issues. I think, to resolve this problem, we should conduct some additional tests to pinpoint where the error is occurring. Previously, we were merely speculating and predicting that it might be a ReadOnly or MOVED error. We should do some debugging and log output.

Yesterday, I made some attempts by sending a large number of set commands to the Redis cluster via a pipeline. Unfortunately, I couldn’t reproduce the issue. I suspect it might be because my cluster is relatively small (only 3 nodes, as I don’t have many computing resources).

Based on our discussion above, we can confirm that #2961 caused the failure. As I mentioned, #2961 only adds more checks for the errors encountered without introducing extra logic—it simply marks the erroneous node. In other words, we’ve established that in this issue, a Redis error was encountered, as hypothesized in the comments above.

We may now need to debug the isBadConn function or add some log output to figure out exactly what’s happening.

go-redis/error.go

Lines 90 to 121 in 1c9309f

	func isBadConn(err error, allowTimeout bool, addr string) bool {
	switch err {
	case nil:
	return false
	case context.Canceled, context.DeadlineExceeded:
	return true
	}
	if isRedisError(err) {
	switch {
	case isReadOnlyError(err):
	// Close connections in read only state in case domain addr is used
	// and domain resolves to a different Redis Server. See #790.
	return true
	case isMovedSameConnAddr(err, addr):
	// Close connections when we are asked to move to the same addr
	// of the connection. Force a DNS resolution when all connections
	// of the pool are recycled
	return true
	default:
	return false
	}
	}
	if allowTimeout {
	if netErr, ok := err.(net.Error); ok && netErr.Timeout() {
	return false
	}
	}
	return true
	}


In the code above, when a context timeout or cancellation, a ReadOnly error, a Moved error, or other unknown errors occur (this is important—did an unknown error arise?), it returns true. Can we add logging, error counts, or some debugging work at these points?

[tim-zhu-reddit]

thanks for the advice, will add log and test it out on Monday

[tim-zhu-reddit]

@monkey92t , added log to each of the cases in isBadConn.
I didn't see any read-only or moved error logs
the vast majority of the errors went into the default case

go-redis/error.go

Lines 109 to 111 in 1c9309f

	default:
	return false
	}

, and the error is redis: nil.

there are also some were not redis errors, reached here,

go-redis/error.go

Lines 113 to 114 in 1c9309f

	if allowTimeout {

the error was like read tcp 10.99.153.22:55070->10.24.94.166:6379: i/o timeout. I think they would count as timeout errors and returned false.

[ndyakov]

If I’m not mistaken (responding from my phone, so can’t check the code right now) the timeouts were not allowed in the said check that was added.

…

On Mon, 17 Mar 2025 at 17:05, tim-zhu-reddit ***@***.***> wrote: @monkey92t <https://github.com/monkey92t> , added log to each of the cases in isBadConn. I didn't see any read-only or moved error logs the vast majority of the errors went into the default case https://github.com/redis/go-redis/blob/1c9309fdc214c77d11953a15113ef6cbcfe49501/error.go#L109-L111 , and the error is redis: nil. there are also some were not redis errors, reached here, https://github.com/redis/go-redis/blob/1c9309fdc214c77d11953a15113ef6cbcfe49501/error.go#L113-L114 the error was like read tcp 10.99.153.22:55070->10.24.94.166:6379: i/o timeout. I think they would count as timeout errors and returned false. — Reply to this email directly, view it on GitHub <#3282 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AALZXMUQNFGGQEW6HIZG5KD2U3QCNAVCNFSM6AAAAABXM5JOK6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOMRZHA3TKMRQGI> . You are receiving this because you were mentioned.Message ID: ***@***.***> [image: tim-zhu-reddit]*tim-zhu-reddit* left a comment (redis/go-redis#3282) <#3282 (comment)> @monkey92t <https://github.com/monkey92t> , added log to each of the cases in isBadConn. I didn't see any read-only or moved error logs the vast majority of the errors went into the default case https://github.com/redis/go-redis/blob/1c9309fdc214c77d11953a15113ef6cbcfe49501/error.go#L109-L111 , and the error is redis: nil. there are also some were not redis errors, reached here, https://github.com/redis/go-redis/blob/1c9309fdc214c77d11953a15113ef6cbcfe49501/error.go#L113-L114 the error was like read tcp 10.99.153.22:55070->10.24.94.166:6379: i/o timeout. I think they would count as timeout errors and returned false. — Reply to this email directly, view it on GitHub <#3282 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AALZXMUQNFGGQEW6HIZG5KD2U3QCNAVCNFSM6AAAAABXM5JOK6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOMRZHA3TKMRQGI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[monkey92t]

Based on the tests we conducted above, I feel very confused. From the discussion above, it can be seen that when the following code is restored, the issue disappears:

if c.opt.ReadOnly && isBadConn(err, false, node.Client.getAddr()) {
    node.MarkAsFailing()
}

However, when you set ReadOnly, RouteRandomly, and RouteByLatency to false, the problem still persists. From this phenomenon, it shouldn’t be the case that simply restoring the above code would resolve it, because it first checks if c.opt.ReadOnly is true. Are we missing something here?

Under normal circumstances, if ReadOnly, RouteRandomly, and RouteByLatency are set to false, the effect of restoring this line of code should be consistent.

---

I’ve reread all the comments again, and I think we should assume that multiple failures together caused this issue. As mentioned above, if it were just a single failure, it would be contradictory.

From the beginning, your operations on the Redis cluster included commands like set, ttl, and mget, with ReadOnly and RouteRandomly enabled as true. This would cause write commands (set/ttl) to be sent to the master, while read commands (mget) would be sent to both the master and slaves. Given your QPS of 350K, in this scenario, when encountering the new error marking introduced in #2961—such as the read ... io/timeout error you described in the comments—the slave would become unavailable. At this point, node.MarkAsFailing() would be triggered, rendering the slave node unavailable for 10 seconds. During this period, set, ttl, and mget commands would all default to the master, causing a surge in traffic to the master node and subsequently triggering more failures.

---

You can perform the following actions: log the locations where node.MarkAsFailing() is triggered as introduced in #2961, along with the error types and associated node information, and compile a statistical summary in the following format:

node(10.24.94.166:6379) - node.Client.getConn(err info...) - count 10
node(10.24.70.53:6379) - WithWriter(err info...) - count 21
node(10.24.92.75:6379) - Read(err info...) - count 28

[tim-zhu-reddit]

However, when you set ReadOnly, RouteRandomly, and RouteByLatency to false, the problem still persists. From this phenomenon, it shouldn’t be the case that simply restoring the above code would resolve it, because it first checks if c.opt.ReadOnly is true. Are we missing something here?

@monkey92t The test I did yesterday was with ReadOnly, RouteRandomly set to true, using the latest master branch as of last week. (here is my logging test changes https://github.com/tim-zhu-reddit/go-redis/pull/1/files). Do you want me to test it again with ReadOnly, RouteRandomly set to false? (with ReadOnly, RouteRandomly set to false, time out errors will be prevalent irregardless of whether the the last line of change in #2961 is reverted or not, that's why I thought it might not be useful.)

[fishy]

@monkey92t I'm not very familiar with redis internals so what I'm saying here might be totally wrong. But from my understanding, let's say we have N redis nodes (1 master, N-1 slaves), what happened here is basically that as long as we have N-1 timeouts from the slaves within 10s, that will mark all the slaves as bad and all requests will go to the master, which will likely overload the master and cause higher latency. (and before we hit N-1 timeouts, any timeout will remove the slave from future requests for 10s, and likely overload other slaves and make them more likely to timeout)

And here, with 350k qps (which is 3.5 million requests for 10s), and usually very small N, this is almost bound to happen.

[ndyakov]

@tim-zhu-reddit if we look again at:

if c.opt.ReadOnly && isBadConn(err, false, node.Client.getAddr()) {
    node.MarkAsFailing()
}

the flag for allowTimeouts in the isBadConn call is false. This means, that timeout errors will trigger the nodes to be marked as failing and probably trigger the flow described by @monkey92t in #3282 (comment) . Maybe the timeout errors were present with the older version as well, but back then they would not result in marking the node as failing.

Can you try increasing the timeouts in the client options and see if the traffic shifts again to the master?

[ndyakov]

@monkey92t I'm not very familiar with redis internals so what I'm saying here might be totally wrong. But from my understanding, let's say we have N redis nodes (1 master, N-1 slaves), what happened here is basically that as long as we have N-1 timeouts from the slaves within 10s, that will mark all the slaves as bad and all requests will go to the master, which will likely overload the master and cause higher latency. (and before we hit N-1 timeouts, any timeout will remove the slave from future requests for 10s, and likely overload other slaves and make them more likely to timeout)

And here, with 350k qps (which is 3.5 million requests for 10s), and usually very small N, this is almost bound to happen.

I am also suspecting something like this happening.

[ndyakov]

@fishy , @tim-zhu-reddit please try with bigger timeouts and report back so we can determine if we should either:

• allow timeouts
• add additional documentation to describe this potential outcome

And mark this as resolved :)

[fishy]

I would propose that we make the behavior introduced by #2961 optional.

Basically, for setup with enough redundancy, #2961 is mostly good, as it helps them to avoid "bad" nodes for a bit.

But, for high through put use cases like ours, any transient network error (it could be timeout or other) will remove one slave for 10s, which will cascade to more timeouts (as every remaining node handling more requests), and quickly leads to all slaves being marked as bad and master node being overloaded.

@ndyakov what do you think?

[ndyakov]

@fishy , @tim-zhu-reddit , @monkey92t I don't think our analysis is correct.

Let's take a step back.

• You are experiencing "normal" latency with the versions before Fix for issues #2959 and #2960 #2961. Those version are more aggressive in marking nodes as failing, since they won't even check the type of the error that occurred, but will mark the node as failing.

• Your load is not balanced with the said versions. Although you are using RouteRandomly which should distribute the read load randomly across slave nodes. Questions:

  • (based on the slots, are the slots balanced? Do you use use hash tags? If so, how? I assume not all keys are in the same slot)
  • Do you have RouteByLatency set? If so, this will make sense that all requests for a given slot are routed to the "closest" node for that slot.

• The changes in Fix for issues #2959 and #2960 #2961 will actually keep more slots "not failing" than before. Since timeouts won't be allowed, I assume they are not the reason why a node is not marked as failing. So I assume you are getting a redis error that is not READONLY or MOVED <same addr>... Let's take LOADING for example

• After the node is not marked as failing, but the command fails with LOADING it will be marked to be retried.

  • I need to do some more investigation here, but retrying a command from the pipeline may slow down the response (increase the latency) for the rest of the commands? I will dig deeper as soon as I have time.

• Could it be that you have a bad node(s) in the cluster and you just never used them that much, since they were marked as failing too early?

• I suspect there is something within the cluster that is the reason for the increased latency. Have you ran the newer version long enough for it to create connections to all nodes? Maybe the latency will reduce as soon as connections are created to the nodes that were previously marked as failing.

In any case, I do think we should gain as much information as possible and document the findings. There may be something in the way the client initiates connections that results in this increase in latency.