Consult zone endpoints while removing unhealthy records

[maksymvavilov]

Why

Currently, we never remove endpoints from the provider when they fail health checks. This is done to prevent NXDomain response. We expect the dns-operator to run in the multicluster environment. This means other controllers could work towards the same root domain and removing endpoints unhealthy endpoints of one of them from the provider might be safe.

What

Consult the DNSRecord.Status.ZoneEndpoints when removing unhealthy records. We want there always to be a chain of endpoints from the root domain to a leaf

How

Two unknowns:

• How to prevent multiple controllers from removing the records at the same time (validation before deletion). We could use a TXT record to mark the intention to delete unhealthy and consult that record when making the decision.
• Do we need to maintain the tree struct for managing the chain of endpoints? The alternative is just a search for a leaf.

Completed after:

1. During the removal of endpoints we consult zone endpoints
2. Added test case for multiple controllers deleting at the same time.

[maksymvavilov]

Also can fix a bug - when determining if reconciling prematurely compare each probe to the dnsrecord status instead of comparing them to each other. Comparing to each other always leads to one false result over 6 possibilities. Comparing to the probe enables us to always have true result