Add config for controlling node ip resolution ordering

[mini-ninja-64]

Hello,

During node discovery, when ZDM proxy finds an entry present in both system.peers and system.local it currently will always prefer node IP addresses returned by system.local. Some Cassandra implementations handle system.local & system.peer tables differently, when the system.peer table contains ALL nodes in the cluster and the ip addresses from system.local are invalid or should not be used then zdm-proxy will currently not work.

This PR adds 2 new config properties (both defaulting to true, to maintain backwards compatability):

• OriginPreferIpFromSystemLocal - Sets origin clusters to prefer IP address from system.local
• TargetPreferIpFromSystemLocal - Sets target clusters to prefer IP address from system.local

These can be used to control the resolution order of IP addresses and would allow for DataStax customers to better migrate from other Cassandra providers.

I am putting this PR out in its current form as an initial conversation starter, I am more than happy to add test coverage for this change, or make other changes, if it is a feature DataStax approves of.

[joao-r-reis]

Hi can you help us understand the issue first? So you're having an issue where the local node is on both system.local and system.peers but the IP address of that node is only valid in system.peers? How can we reproduce this scenario? Which Cassandra implementations/providers implement this logic?

[mini-ninja-64]

Hi can you help us understand the issue first? So you're having an issue where the local node is on both system.local and system.peers but the IP address of that node is only valid in system.peers? How can we reproduce this scenario? Which Cassandra implementations/providers implement this logic?

Of course! Thank you so much for looking into this, apologies for my delay in response.

As far as I am aware, "AWS Keyspaces" is the only Cassandra implementation with this quirk, they currently present all IPs in the system.local table as 127.0.0.1, fwiw they have a small note about this behaviour in their documentation.

Because of this to get the external IP of a current node in AWS Keyspaces you must use the system.peers table which also includes the current host. Currently AWS Keyspaces when used with ZDM Proxy will fail as during node discovery the 127.0.0.1 IP from system.local will replace the IP used for a node in ZDM Proxy, as 127.0.0.1 is not likely to be a valid Cassandra ZDM Proxy will fail to resolve any queries for the node.