When helm install single-node etcd, delete the etcd pod etcd fails to start

[CharesLee]

What happened?

etcd not runing

What did you expect to happen?

etcd running

How can we reproduce it (as minimally and precisely as possible)?

Please look at bitnami/charts#16585

Anything else we need to know?

No response

Etcd version (please run commands below)

bitnami/etcd 8.5.11
etcd 3.5.6

Etcd configuration (command line flags or environment variables)

paste your configuration here

Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

$ etcdctl member list -w table
# paste output here

$ etcdctl --endpoints=<member list> endpoint status -w table
# paste output here

Relevant log output

No response

[jmhbnz]

Hey @CharesLee - I wasn't able to reproduce this, the steps I followed were:

# First deploy
helm repo add bitnami https://charts.bitnami.com/bitnami
helm install etcd bitnami/etcd --version 8.5.11 -f values.yaml

# Wait for pod to become ready

# Check etcdctl member list to confirm etcd running
kubectl exec -it etcd-0 etcdctl member list

# Delete the successfully running pod
kubectl delete po etcd-0

# Confirm new pod runs successfully
kubectl get po
kubectl exec -it etcd-0 etcdctl member list

I used the exact values.yaml you provided in the linked issue.

Note I did use kubernetes version 1.26 as 1.23 is no longer supported. If you are still having issues after upgrading to a supported kubernetes release and the latest etcd 3.5.x patch version then please provide precise steps we can use to reproduce the issue.

[CharesLee]

ok, thanks for the answer; as I said before this problem doesn't come up every time, so it's weird.
But indeed we have encountered this situation in several environments (especially in the privatized deployment to the customer environment, this problem is very distressing)
Or can you tell me under what circumstances etcd will keep outputting this kind of log information when it starts

{"level":"warn","ts":"2023-05-11T09:52:34.517Z","caller":"etcdserver/server.go:2075","msg":"failed to publish local member to cluster through raft","local-member-id":"7cdfc1f3de691ee2","local-member-attributes":"{Name:etcd-0 ClientURLs:[http://etcd-0.etcd-headless.cguarantee.svc.cluster.local:2379 http://etcd.cguarantee.svc.cluster.local:2379]}","request-path":"/0/members/7cdfc1f3de691ee2/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"}

Finally, I also plan to update the latest 3.5.x version of etcd to test to see if this problem has been resolved

[jmhbnz]

There can be a variety of reasons why requests can time out, network issues, server load, disk latency etc. Please proceed with an upgrade to a supported kubernetes release and ideally the latest etcd 3.5 patch release.

If you can still reproduce the issue afterwards please reply with the full startup log from the etcd instance along with the configuration file or parameters used and we can take another look 🙏🏻

[chaochn47]

@CharesLee Could you please also turn the log level to Debug when you reproduce the issue? This should help narrow down which step etcd is not progressing.

For example, I am interested in

etcd/server/etcdserver/server.go

Line 2219 in c919780

s.lg.Debug("applyEntryNormal", zap.Stringer("V2request", rp))

I am surprised that publish local member to cluster through raft will timeout in a single node cluster :P