ETCD peer discovery error when trying to automatically clean up the node

[rtuk]

Describe the bug

With ETCD peer discovery enabled, trying to clean up a node that left the cluster results in an error:

2024-11-25 17:16:24.074332+01:00 [error] <0.3336.0> ** Generic server rabbit_peer_discovery_cleanup terminating , ** Last message in was check_cluster, ** When Server state == {state,60,false, {interval,#Ref<0.2371737395.93061121.59530>}},
** Reason for termination ==, ** {badarg, [{erlang,'--', [[rabbit@caa95500c223],rabbit@d0bee83164eb], [{error_info,#{module => erl_erts_errors}}]}, {lists,subtract,2,[{file,"lists.erl"},{line,254}]}, {rabbit_peer_discovery_cleanup,maybe_cleanup,2, [{file,"rabbit_peer_discovery_cleanup.erl"},{line,244}]}, {rabbit_peer_discovery_cleanup,handle_call,3, 
[{file,"rabbit_peer_discovery_cleanup.erl"},{line,129}]}, {gen_server,try_handle_call,4,
[{file,"gen_server.erl"},{line,2381}]}, {gen_server,handle_msg,6,[{file,"gen_server.erl"},
{line,2410}]}, {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,329}]}]}, ** Client <0.3782.0> stacktrace, ** [{gen,do_call,4,[{file,"gen.erl"},{line,260}]}, {gen_server,call,2,[{file,"gen_server.erl"},{line,1138}]}, {rabbit_peer_discovery_cleanup,check_cluster,0, [{file,"rabbit_peer_discovery_cleanup.erl"}, {line,62}]}],

2024-11-25 17:16:24.075322+01:00 [error] <0.3336.0>   crasher:, initial call: rabbit_peer_discovery_cleanup:init/1, pid: <0.3336.0>, registered_name: rabbit_peer_discovery_cleanup, 
exception error: bad argument, in operator  --/2, called as [rabbit@caa95500c223] -- rabbit@d0bee83164eb, 
in call from lists:subtract/2 (lists.erl, line 254), in call from rabbit_peer_discovery_cleanup:maybe_cleanup/2 (rabbit_peer_discovery_cleanup.erl, line 244),
in call from rabbit_peer_discovery_cleanup:handle_call/3 (rabbit_peer_discovery_cleanup.erl, line 129), in call from gen_server:try_handle_call/4 (gen_server.erl, line 2381), in call from gen_server:handle_msg/6 (gen_server.erl, line 2410), 

ancestors: [rabbit_peer_discovery_common_sup,<0.751.0>], message_queue_len: 0, messages: [], links: [<0.752.0>], dictionary: [], trap_exit: false, status: running, heap_size: 10958, stack_size: 29, reductions: 21701, neighbours:,
rabbitmq.1.twjfspvpbt2m@docker-desktop    | 2024-11-25 17:16:24.076012+01:00 [error] <0.752.0>     supervisor: {local,rabbit_peer_discovery_common_sup}, errorContext: child_terminated, reason: {badarg, [{erlang,'--', [[rabbit@caa95500c223],rabbit@d0bee83164eb], [{error_info,#{module => erl_erts_errors}}]}, {lists,subtract,2,[{file,"lists.erl"},{line,254}]}, {rabbit_peer_discovery_cleanup,maybe_cleanup,2, [{file,"rabbit_peer_discovery_cleanup.erl"},{line,244}]}, {rabbit_peer_discovery_cleanup,handle_call,3, [{file,"rabbit_peer_discovery_cleanup.erl"},{line,129}]}, {gen_server,try_handle_call,4, [{file,"gen_server.erl"},{line,2381}]}, {gen_server,handle_msg,6, [{file,"gen_server.erl"},{line,2410}]}, {proc_lib,init_p_do_apply,3, [{file,"proc_lib.erl"},{line,329}]}]}, offender: [{pid,<0.3336.0>}, {id,rabbit_peer_discovery_cleanup}, {mfargs,{rabbit_peer_discovery_cleanup,start_link,[]}}, {restart_type,permanent}, {significant,false}, {shutdown,300000}, {child_type,worker}],

Reproduction steps

start a local etcd "cluster":
/usr/local/bin/etcd
--name etcd-local
--initial-advertise-peer-urls http://0.0.0.0:2380
--listen-peer-urls http://0.0.0.0:2380,http://0.0.0.0:7001
--listen-client-urls http://0.0.0.0:2379,http://0.0.0.0:4001
--advertise-client-urls http://0.0.0.0:2379,http://0.0.0.0:4001,http://127.0.0.1:2379,http://127.0.0.1:4001
--initial-cluster-token local-etcd-cluster
--initial-cluster etcd-local=http://0.0.0.0:2380
--initial-cluster-state new

run rabbitmq in docker swarm, with 3 replicas and the following cluster_formation config:

cluster_formation.peer_discovery_backend = etcd
cluster_formation.etcd.node_ttl = 60
cluster_formation.node_cleanup.only_log_warning = false
cluster_partition_handling = autoheal
cluster_formation.etcd.endpoints.1 = docker.for.mac.localhost:2379
cluster_formation.etcd.cluster_name = rabbitmq

after the cluster is started, kill a node

Expected behavior

killed docker task is started again and joins the cluster
"failed" node is removed from the cluster instead of being shown as "Node not running"

Additional context

No response

[michaelklishin]

I don't understand what's being asked here.

The exception comes from this line as [rabbit@caa95500c223] -- rabbit@d0bee83164eb, where the second argument must be a list for lists:--/2 to accept it.

I don't see how this can prevent the node from re-joining the cluster. Most likely this simply creates log noise on other nodes.

[michaelklishin]

At least in main and v4.0.x, https://github.com/rabbitmq/rabbitmq-server/blob/main/deps/rabbitmq_peer_discovery_common/src/rabbit_peer_discovery_cleanup.erl#L303-L319 seems correct and should return a list of nodes.

Either this is on an older version, or I cannot think of a case where a single node (an atom) would be used.

@rtuk you have left out the most important piece of information: what RabbitMQ is this on?

[rtuk]

sorry about that, I ran into this issue when testing the upgrade to 4.0.4

[michaelklishin]

A simpler way to reproduce:

# rabbitmq.conf

# Pick any peer discovery mechanism with a dynamically fetched list of nodes:
# etcd, Consul, AWS, Kubernetes and configure it

# opt-in to have unavailable nodes forcefully removed from the cluster
cluster_formation.node_cleanup.only_log_warning = false

and then stop any of the cluster nodes.

For https://github.com/rabbitmq/rabbitmq-server/blob/main/deps/rabbitmq_peer_discovery_common/src/rabbit_peer_discovery_cleanup.erl#L244 and https://github.com/rabbitmq/rabbitmq-server/blob/main/deps/rabbitmq_peer_discovery_common/src/rabbit_peer_discovery_cleanup.erl#L303-L319
to result in a RHS argument being an atom, as in the exception above:

[rabbit@caa95500c223] -- rabbit@d0bee83164eb

This function must return a single node value. Which it only does if the input is a single atom and it should not be in main and v4.0.x.

[michaelklishin]

#12809

[michaelklishin]

Per discussion with another core team member: indeed, the "single value" results of peer discovery is a special case introduced intentionally to allow for a modified seeding (see RabbitMQ 4.0 release notes for details) by certain backends.

The cleanup module simply wasn't updated to account for that, hence #12809.

[michaelklishin]

@rtuk here are 4.0.5 preview packages you can use to test the proposed fix.

For an OCI image, see https://hub.docker.com/r/pivotalrabbitmq/rabbitmq/tags. It was updated automatically by Actions.

[rtuk]

Thank you, with this version the cleanup works as expected again!
Looking forward to the release

[michaelklishin]

@rtuk a new patch release will take about 3-4 weeks (we produce patch releases roughly once a month) but you have all the artifacts from rabbitmq/server-packages/releases and a publicly available OCI. I understand that it is not idea but for development environments it should be enough to unblock your work.