Bug: quorum management has potential deadlock

[james-lawrence]

looks specifically gcp related. but basically gcp rebuilds a server and due to the number of servers it'll both fail to add the rebuilt machine due to duplicate names and fail to remove the old node from quorum because it lacks voters.

May 24 21:47:51 esuite-prod-rz7q bw[3436]: [AGENT] 2024/05/24 21:47:51 leader.go:102: failed to remove peer, not enough voters
May 24 21:47:51 esuite-prod-rz7q bw[3436]: [AGENT] 2024/05/24 21:47:51 leader.go:29: peers unstable, will refresh in 1s
May 24 21:47:52 esuite-prod-rz7q bw[3436]: [AGENT] 2024/05/24 21:47:52 leader.go:24: leader update invoked: 0xc00043c588 - 1h0m0s
May 24 21:47:52 esuite-prod-rz7q bw[3436]: [AGENT] 2024/05/24 21:47:52 leader.go:83: failed to add peer (raft.Server) {
May 24 21:47:52 esuite-prod-rz7q bw[3436]:  Suffrage: (raft.ServerSuffrage) Voter,
May 24 21:47:52 esuite-prod-rz7q bw[3436]:  ID: (raft.ServerID) (len=64) "05f4531cdbdf9a7cbb340a2837a6eb00425b79de34607612dd8f4b3f04b99154",
May 24 21:47:52 esuite-prod-rz7q bw[3436]:  Address: (raft.ServerAddress) (len=16) "10.127.0.10:2000"
May 24 21:47:52 esuite-prod-rz7q bw[3436]: }
May 24 21:47:52 esuite-prod-rz7q bw[3436]:  ([]raft.Server) (len=3 cap=3) {
May 24 21:47:52 esuite-prod-rz7q bw[3436]:  (raft.Server) {
May 24 21:47:52 esuite-prod-rz7q bw[3436]:   Suffrage: (raft.ServerSuffrage) Voter,
May 24 21:47:52 esuite-prod-rz7q bw[3436]:   ID: (raft.ServerID) (len=64) "b0cbd8de9bc21bcf7e3fe379ff957fa36482a9b8dadec33ba0438fe8169085d9",
May 24 21:47:52 esuite-prod-rz7q bw[3436]:   Address: (raft.ServerAddress) (len=15) "10.127.0.6:2000"
May 24 21:47:52 esuite-prod-rz7q bw[3436]:  },
May 24 21:47:52 esuite-prod-rz7q bw[3436]:  (raft.Server) {
May 24 21:47:52 esuite-prod-rz7q bw[3436]:   Suffrage: (raft.ServerSuffrage) Voter,
May 24 21:47:52 esuite-prod-rz7q bw[3436]:   ID: (raft.ServerID) (len=64) "49602bf7155d99118e514479815d9950c9040fa1af6b1b17f042d0cc01ac64b4",
May 24 21:47:52 esuite-prod-rz7q bw[3436]:   Address: (raft.ServerAddress) (len=15) "10.127.0.9:2000"
May 24 21:47:52 esuite-prod-rz7q bw[3436]:  },
May 24 21:47:52 esuite-prod-rz7q bw[3436]:  (raft.Server) {
May 24 21:47:52 esuite-prod-rz7q bw[3436]:   Suffrage: (raft.ServerSuffrage) Nonvoter,
May 24 21:47:52 esuite-prod-rz7q bw[3436]:   ID: (raft.ServerID) (len=64) "43649f449abfcb43e559d83ac5822cf7fd87ac03a7c3099de5b92491698c74f1",
May 24 21:47:52 esuite-prod-rz7q bw[3436]:   Address: (raft.ServerAddress) (len=16) "10.127.0.10:2000"
May 24 21:47:52 esuite-prod-rz7q bw[3436]:  }
May 24 21:47:52 esuite-prod-rz7q bw[3436]: }
May 24 21:47:52 esuite-prod-rz7q bw[3436]:  found duplicate address in configuration: 10.127.0.10:2000
May 24 21:47:53 esuite-prod-rz7q bw[3436]: [AGENT] 2024/05/24 21:47:53 leader.go:102: failed to remove peer, not enough voters
May 24 21:47:53 esuite-prod-rz7q bw[3436]: [AGENT] 2024/05/24 21:47:53 leader.go:29: peers unstable, will refresh in 1s