Investigate DNSPolicy supporting advanced load balancing without OCM

[maleck13]

What

Understand whether it is possible to support the advanced multi-cluster load balancing options provided by DNSPolicy without the need for the single view provided by OCM. Really what we are trying to do here is have two DNS controllers running independently (IE on different clusters) but managing a DNSPolicy that targets a gateway that shares a host and zone

How

There are three key parts to this

1. defining the records names so that the right record set is created regardless of the number of instances
2. coming up with a way to manage a "dead" cluster / orphaned records
3. coming up with a way to define a "leader" that is responsible for shared records

Phase 1 idea

Get to the point where 2 or more clusters can share a zone and a host in a single DNS provider but are not worried about the number of requests to the DNSprovider and there is no leader or orphan record clean up

We currently set up the shared load balancer records based on the "central ocm gateway" namespace and name in the form of a short code . We then use the clustername and gateway name to create a unique short code for records relating to each individual instance of that gateway.

To allow this to work without OCM in the mix, we want to explore the following:

Each instance of the controller is started with a lb-prefix flag as these records are the same no matter which cluster they are created from, this should allow the shared records to be created and not overwritten.

Each gateway instance has a label or annotation that is the clusterID. This is unique to the cluster and should not be shared.
This will be used to set up the "gateway" records under the shared lb structure

Known Caveats

• Deleting the DNSRecord or DNSPolicy will delete all of the shared records plus the individual gateway record for that cluster

Mitigation, that once the DNSRecord resource was gone, soon afterwards the other controller would recreate this structure and the remaining gateway records

• If one of the controller fails for some reason and does not recover, you will end up with "orphaned" records
  Mitigation: none currently, this would require a manual process to clean up

[philbrookes]

This issue is stale because it has been open for 60 days with no activity.

[maleck13]

closing as done
Kuadrant/architecture#55