"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"

[Tejaswini5327]

Bug report criteria

• This bug report is not security related, security issues should be disclosed privately via [email protected].
• This is not a support request, support requests should be raised in the etcd discussion forums.
• You have read the etcd bug reporting guidelines.
• Existing open issues along with etcd frequently asked questions have been checked and this is not a duplicate.

What happened?

When trying to do scale pods , successfully scaled po-2, trying to scale pod-1 then checking etcdctl member list in pod-0 , giving error as {"level":"warn","ts":"2023-07-26T03:59:26.143+0200","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-b1400e47-e7c8-4037-9a76-00e24ef95743/eric-data-distributed-coordinator-ed.eda:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
Error: context deadline exceeded
command terminated with exit code 1
and checked the cluster heakth using etcdctl endpoint health , cluster is not healthy.

What did you expect to happen?

1. code = DeadlineExceeded desc = context deadline exceeded should not come and any reason for that.
2. in that if cluster is unhealthy, how to recover the cluster from this situation.

How can we reproduce it (as minimally and precisely as possible)?

after scaling one pod, when trying to check etcdctl member list , then giving this error.

Anything else we need to know?

No response

Etcd version (please run commands below)

$ etcd --version
3.4.16

Etcd configuration (command line flags or environment variables)

paste your configuration here

Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

$ etcdctl member list -w table
# paste output here

$ etcdctl --endpoints=<member list> endpoint status -w table
# paste output here

Relevant log output

bash-4.4$ etcdctl endpoint health\cf1\highlight2 
\par \cf0\highlight0\{"level":"warn","ts":"2023-07-31T10:05:31.487+0200","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-ea41b178-5173-4807-82cb-dac879f61911/eric,"attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"\}\cf1\highlight2 
\par \cf0\highlight0 etcd.eda:2379 is unhealthy: failed to commit proposal: context deadline exceeded\cf1\highlight2 
\par \cf0\highlight0 Error: unhealthy cluster\cf1\highlight2 
\par \cf0\highlight0 bash-4.4$ etcdctl endpoint health\cf1\highlight2

[jmhbnz]

Hey @Tejaswini5327 - Etcd version 3.4.16 is now a couple years old and there have been many changes and fixes since that release. Are you able to confirm the issue also occurs on a more recent release, ideally 3.5.6+?

If the issue persists, please provide clear step by step instructions on how to recreate it so we can investigate. Thanks.

[Tejaswini5327]

Hi @jmhbnz ,

We didnt see any issue in etcd 3.5.5 version we are using, we saw issue in 3.4.16 only as of now.