Member Client URLs Advertisement Behavior

[jabellard]

Per the operations guide, each member of a cluster should advertise its own cluster-accessible client URLs. However, from some testing, I noticed that having each member of the cluster advertise not only its own client URLs, but also the client URLs of all the other members of the cluster works just fine.

Given that this is not the recommended setup per the operations guide, I wonder if this type of configuration could potentially open up the door to insidious, undefined behavior and should be avoided.

I'm not an etcd expert, so I'm asking these questions out of curiosity for my own learning.

[jabellard]

@jmhbnz , @ahrtr , @serathius : I'd love to hear your thoughts on this.

[jabellard]

@wenjiaswe

[ahrtr]

--advertise-client-urls just advertise each member's client URLs for clients to connect to. If you configure wrong URLs, then the only issue is that it may mislead the clients. Other than this, there is no other security issue.

For example, assuming you have an etcd cluster something like below,

$ etcdctl member list -w table
+------------------+---------+--------+------------------------+------------------------+------------+
|        ID        | STATUS  |  NAME  |       PEER ADDRS       |      CLIENT ADDRS      | IS LEARNER |
+------------------+---------+--------+------------------------+------------------------+------------+
| 8211f1d0f64f3269 | started | infra1 | http://127.0.0.1:12380 |  http://127.0.0.1:2379 |      false |
| 91bc3c398fb3c146 | started | infra2 | http://127.0.0.1:22380 | http://127.0.0.1:22379 |      false |
| fd422379fda50e48 | started | infra3 | http://127.0.0.1:32380 | http://127.0.0.1:32379 |      false |
+------------------+---------+--------+------------------------+------------------------+------------+

$ etcdctl endpoint status -w table --cluster
+------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|        ENDPOINT        |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|  http://127.0.0.1:2379 | 8211f1d0f64f3269 |  3.5.16 |   98 kB |     false |      false |         2 |          8 |                  8 |        |
| http://127.0.0.1:22379 | 91bc3c398fb3c146 |  3.5.16 |   98 kB |      true |      false |         2 |          8 |                  8 |        |
| http://127.0.0.1:32379 | fd422379fda50e48 |  3.5.16 |   98 kB |     false |      false |         2 |          8 |                  8 |        |
+------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

Now if you incorrectly configured --advertise-client-urls http://127.0.0.1:2379,http://192.168.1.10:2379 for the first member (etcd1), then you will have the same etcd cluster, the only issue is the member etcd1 advertises a wrong client URL,

$ etcdctl member list -w table
+------------------+---------+--------+------------------------+------------------------------------------------+------------+
|        ID        | STATUS  |  NAME  |       PEER ADDRS       |                  CLIENT ADDRS                  | IS LEARNER |
+------------------+---------+--------+------------------------+------------------------------------------------+------------+
| 8211f1d0f64f3269 | started | infra1 | http://127.0.0.1:12380 | http://127.0.0.1:2379,http://192.168.1.10:2379 |      false |
| 91bc3c398fb3c146 | started | infra2 | http://127.0.0.1:22380 |                         http://127.0.0.1:22379 |      false |
| fd422379fda50e48 | started | infra3 | http://127.0.0.1:32380 |                         http://127.0.0.1:32379 |      false |
+------------------+---------+--------+------------------------+------------------------------------------------+------------+

$ etcdctl endpoint status -w table --cluster
{"level":"warn","ts":"2024-11-15T09:39:52.165446Z","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:63","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x140000d0000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
Failed to get the status of endpoint http://192.168.1.10:2379 (context deadline exceeded)
+------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|        ENDPOINT        |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|  http://127.0.0.1:2379 | 8211f1d0f64f3269 |  3.5.16 |   98 kB |     false |      false |         2 |          8 |                  8 |        |
| http://127.0.0.1:22379 | 91bc3c398fb3c146 |  3.5.16 |   98 kB |     false |      false |         2 |          8 |                  8 |        |
| http://127.0.0.1:32379 | fd422379fda50e48 |  3.5.16 |   98 kB |      true |      false |         2 |          8 |                  8 |        |
+------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

[jabellard]

@ahrtr , thanks for following up on this.