Best way to rename an etcd cluster without downtime?

[ronwildt]

Looking for the best way to rename an etcd cluster and if possible do so with minimal or no downtime. Have searched extensively but doesn't appear to be an issue I can find an answer for, hoping the community can help! Thanks!

[jmhbnz]

Hey @ronwildt - Thanks for your question. Can you please confirm your cluster size? This will influence how safely individual nodes can be updated.

[ronwildt]

Hi @jmhbnz - We have multiple but most are 5 node clusters.

[jmhbnz]

Thanks for the clarification, ok so assuming it is nodes you want to rename and for a cluster with some fault tolerance, i.e. 3+ nodes I was able to do the following:

Step 1- Initial cluster startup

Node config files

name: node1
force-new-cluster: false
initial-advertise-peer-urls: http://localhost:2381
listen-peer-urls: "http://localhost:2381"
advertise-client-urls: "http://localhost:2377"
listen-client-urls: "http://localhost:2377"
initial-cluster: node1=http://localhost:2381,node2=http://localhost:2382,node3=http://localhost:2383

name: node2
force-new-cluster: false
initial-advertise-peer-urls: http://localhost:2382
listen-peer-urls: "http://localhost:2382"
advertise-client-urls: "http://localhost:2378"
listen-client-urls: "http://localhost:2378"
initial-cluster: node1=http://localhost:2381,node2=http://localhost:2382,node3=http://localhost:2383

name: node3
force-new-cluster: false
initial-advertise-peer-urls: http://localhost:2383
listen-peer-urls: "http://localhost:2383"
advertise-client-urls: "http://localhost:2379"
listen-client-urls: "http://localhost:2379"
initial-cluster: node1=http://localhost:2381,node2=http://localhost:2382,node3=http://localhost:2383

Launch cluster nodes and check member list

./etcd --config-file node1.yaml&
./etcd --config-file node2.yaml&
./etcd --config-file node3.yaml&

 james  ~  Documents  etcd  bin   main                                                                                                                                        06:15:46 
 ➜ ./etcdctl member list --endpoints http://localhost:2377,http://localhost:2378,http://localhost:2379 -w table
+------------------+---------+-------+-----------------------+-----------------------+------------+
|        ID        | STATUS  | NAME  |      PEER ADDRS       |     CLIENT ADDRS      | IS LEARNER |
+------------------+---------+-------+-----------------------+-----------------------+------------+
| 324473db0474a678 | started | node3 | http://localhost:2383 | http://localhost:2379 |      false |
| 492733e293f5a27b | started | node1 | http://localhost:2381 | http://localhost:2377 |      false |
| 7e3bd17c66e004e8 | started | node2 | http://localhost:2382 | http://localhost:2378 |      false |
+------------------+---------+-------+-----------------------+-----------------------+------------+

Step 2 - Identify leader

With our cluster running, let's confirm which node is currently the leader. We want to leave this node for last.

 james  ~  Documents  etcd  bin   main                                                                                                                                        06:19:43 
 ➜ ./etcdctl endpoint status --endpoints http://localhost:2377,http://localhost:2378,http://localhost:2379 -w table
+-----------------------+------------------+---------------+-----------------+---------+----------------+-----------+------------+-----------+------------+--------------------+--------+
|       ENDPOINT        |        ID        |    VERSION    | STORAGE VERSION | DB SIZE | DB SIZE IN USE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+-----------------------+------------------+---------------+-----------------+---------+----------------+-----------+------------+-----------+------------+--------------------+--------+
| http://localhost:2377 | 492733e293f5a27b | 3.6.0-alpha.0 |           3.6.0 |   20 kB |          16 kB |      true |      false |         2 |          9 |                  9 |        |
| http://localhost:2378 | 7e3bd17c66e004e8 | 3.6.0-alpha.0 |           3.6.0 |   20 kB |          16 kB |     false |      false |         2 |          9 |                  9 |        |
| http://localhost:2379 | 324473db0474a678 | 3.6.0-alpha.0 |           3.6.0 |   20 kB |          16 kB |     false |      false |         2 |          9 |                  9 |        |
+-----------------------+------------------+---------------+-----------------+---------+----------------+-----------+------------+-----------+------------+--------------------+--------+

Step 3 - Rename a node via remove and re-add

# Stop etcd process for node

# Remove a member that is not a leader, in my case node3
./etcdctl --endpoints http://localhost:2377,http://localhost:2378,http://localhost:2379 member remove 324473db0474a678

# Add an unstarted new member with the desired new name
./etcdctl --endpoints http://localhost:2377,http://localhost:2378,http://localhost:2379 member add newname3 --peer-urls=http://localhost:2383

Update configuration file for node3 to new name in both name and initial-cluster fields, and also now set the initial-cluster-state to existing.

name: newname3
force-new-cluster: false
initial-advertise-peer-urls: http://localhost:2383
listen-peer-urls: "http://localhost:2383"
advertise-client-urls: "http://localhost:2379"
listen-client-urls: "http://localhost:2379"
initial-cluster: node1=http://localhost:2381,node2=http://localhost:2382,newname3=http://localhost:2383
initial-cluster-state: existing

Now we can start the new member and check cluster status:

./etcd --config-file node3.yaml&

 james  ~  Documents  etcd  bin   main                                                                                                                                        06:41:55 
 ➜ ./etcdctl member list --endpoints http://localhost:2377,http://localhost:2378,http://localhost:2379 -w table
+------------------+---------+----------+-----------------------+-----------------------+------------+
|        ID        | STATUS  |   NAME   |      PEER ADDRS       |     CLIENT ADDRS      | IS LEARNER |
+------------------+---------+----------+-----------------------+-----------------------+------------+
| 492733e293f5a27b | started |    node1 | http://localhost:2381 | http://localhost:2377 |      false |
| 7e3bd17c66e004e8 | started |    node2 | http://localhost:2382 | http://localhost:2378 |      false |
| 89241d1e713c88c4 | started | newname3 | http://localhost:2383 | http://localhost:2379 |      false |
+------------------+---------+----------+-----------------------+-----------------------+------------+

 james  ~  Documents  etcd  bin   main                                                                                                                                        06:41:44 
 ➜ ./etcdctl endpoint status --endpoints http://localhost:2377,http://localhost:2378,http://localhost:2379 -w table
+-----------------------+------------------+---------------+-----------------+---------+----------------+-----------+------------+-----------+------------+--------------------+--------+
|       ENDPOINT        |        ID        |    VERSION    | STORAGE VERSION | DB SIZE | DB SIZE IN USE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+-----------------------+------------------+---------------+-----------------+---------+----------------+-----------+------------+-----------+------------+--------------------+--------+
| http://localhost:2377 | 492733e293f5a27b | 3.6.0-alpha.0 |           3.6.0 |   20 kB |          16 kB |      true |      false |         2 |         12 |                 12 |        |
| http://localhost:2378 | 7e3bd17c66e004e8 | 3.6.0-alpha.0 |           3.6.0 |   20 kB |          16 kB |     false |      false |         2 |         12 |                 12 |        |
| http://localhost:2379 | 89241d1e713c88c4 | 3.6.0-alpha.0 |           3.6.0 |   20 kB |          16 kB |     false |      false |         2 |         12 |                 12 |        |
+-----------------------+------------------+---------------+-----------------+---------+----------------+-----------+------------+-----------+------------+--------------------+--------+

Cluster is still healthy, there has been no cluster downtime, and one member has been renamed.

You would need to rinse and repeat a similar process for each member in the cluster.

[ronwildt]

@jmhbnz Thanks James! Curious if this process is documented anywhere 'official'? I've searched but couldn't find anything on this. If so, please post the link here but either way, thanks for your feedback on the process! :)

[jmhbnz]

I've adapted the instructions above from this docs page as a base: https://etcd.io/docs/v3.5/tutorials/how-to-deal-with-membership

While the desired outcome is renaming a node, under the hood from what I can see we just need to be following the member add remove process, adjusting name when we re-add.

I do think we could add some docs to specifically talk about "renaming" in our ops guide or faq though. I've raised etcd-io/website#715 to track this.

[ronwildt]

Thanks James! @jmhbnz Thanks for the doc reference. Hope you have a wonderful weekend!

[ahrtr]

I am not sure I understand "rename an etcd cluster". Do you mean you want to change the value to flag --initial-cluster-token?