Report healtchecks to status

[maksymvavilov]

Reports health checks into the status of DNSRecord.
It also modifies the Ready status. The logic is:

• If all checks pass - the record is healthy and ready
• If we have failing and healthy probes - the record is not healthy but ready
• If all checks fail - the record is not healthy and not ready

[maksymvavilov]

this will fail e2e now so removing. Will be brought back in a next PR

[maleck13]

tried with the following setup:

---
kind: Gateway
apiVersion: gateway.networking.k8s.io/v1
metadata:
  name: external
spec:
  gatewayClassName: istio
  listeners:
    - name: t1a
      port: 80
      hostname: 't1a.cb.hcpapps.net'
      protocol: HTTP
    - name: t1b
      port: 80
      hostname: 't1b.cb.hcpapps.net'
      protocol: HTTP
    - name: t1c
      port: 80
      hostname: 't1c.cb.hcpapps.net'
      protocol: HTTP
    - name: t1d
      port: 80
      hostname: 't1d.cb.hcpapps.net'
      protocol: HTTP

---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: toystore
  labels:
    app: toystore
spec:
  parentRefs:
    - name: external
  hostnames:
    [
      't1a.cb.hcpapps.net',
      't1b.cb.hcpapps.net',
      't1c.cb.hcpapps.net',
      't1d.cb.hcpapps.net',
    ]
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: '/toy'
          method: GET
        - path:
            type: Exact
            value: '/admin/toy'
          method: POST
        - path:
            type: Exact
            value: '/admin/toy'
          method: DELETE
        - path:
            type: Exact
            value: '/health'
          method: GET
      backendRefs:
        - name: toystore
          port: 80
---
apiVersion: kuadrant.io/v1alpha1
kind: DNSPolicy
metadata:
  name: dnspolicy-t1a
  labels:
    policy: 'gateway'
spec:
  loadBalancing:
    weight: 125
    geo: GEO-EU
    defaultGeo: true
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: external
    sectionName: t1a
  providerRefs:
    - name: aws-credentials
  healthCheck:
    path: /health
    protocol: HTTP
    failureThreshold: 3
    interval: 30s
---
apiVersion: kuadrant.io/v1alpha1
kind: DNSPolicy
metadata:
  name: dnspolicy-t1b
  labels:
    policy: 'gateway'
spec:
  loadBalancing:
    weight: 125
    geo: GEO-EU
    defaultGeo: true
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: external
    sectionName: t1b
  providerRefs:
    - name: aws-credentials
  healthCheck:
    path: /healthz
    protocol: HTTP
    failureThreshold: 3
    interval: 30s
---
apiVersion: kuadrant.io/v1alpha1
kind: DNSPolicy
metadata:
  name: dnspolicy
  labels:
    policy: 'gateway'
spec:
  loadBalancing:
    weight: 125
    geo: GEO-EU
    defaultGeo: true
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: external
  providerRefs:
    - name: aws-credentials
  healthCheck:
    path: /health
    protocol: HTTP
    failureThreshold: 3
    interval: 30s

This resulted in

k get dnshealthcheckprobe -n t1 -o=wide
NAME                                                                                  HEALTHY   LAST CHECKED
external-t1a-aef422ea1007f49b19e8ca537252a20b-411358573.us-east-1.elb.amazonaws.com   true      18s
external-t1b-aef422ea1007f49b19e8ca537252a20b-411358573.us-east-1.elb.amazonaws.com   false     20s
external-t1c-aef422ea1007f49b19e8ca537252a20b-411358573.us-east-1.elb.amazonaws.com   true      14s
external-t1d-aef422ea1007f49b19e8ca537252a20b-411358573.us-east-1.elb.amazonaws.com   true      13s

k get dnsrecord.kuadrant.io -n t1 -o=wide
NAME           READY
external-t1a   True
external-t1b   False
external-t1c   True
external-t1d   True

k get dnspolicy -n t1 -o=wide
NAME            ACCEPTED   ENFORCED   TARGETREFKIND   TARGETREFNAME   AGE
dnspolicy       True       True       Gateway         external        2m30s
dnspolicy-t1a   True       True       Gateway         external        2m31s
dnspolicy-t1b   True       False      Gateway         external        2m30s

as expected and all but t1b records were present in the provider

Updating the dnspolicy-t1b dnspolicy to have a valid endpoint resulted in

k get dnspolicy -n t1 -o=wide
NAME            ACCEPTED   ENFORCED   TARGETREFKIND   TARGETREFNAME   AGE
dnspolicy       True       True       Gateway         external        6m39s
dnspolicy-t1a   True       True       Gateway         external        6m40s
dnspolicy-t1b   True       True       Gateway         external        6m39s

as expected and t1b plus the others records were correct in the provider

Scaling down the workload resulted in the records remaining in place and all the health checks and dnspolicies becoming unhealthy

external-t1a-aef422ea1007f49b19e8ca537252a20b-411358573.us-east-1.elb.amazonaws.com   false     21s
external-t1b-aef422ea1007f49b19e8ca537252a20b-411358573.us-east-1.elb.amazonaws.com   false     19s
external-t1c-aef422ea1007f49b19e8ca537252a20b-411358573.us-east-1.elb.amazonaws.com   false     16s
external-t1d-aef422ea1007f49b19e8ca537252a20b-411358573.us-east-1.elb.amazonaws.com   false     15s

although it took a while the DNSPolicy to show as not enforced they eventually did

NAME            ACCEPTED   ENFORCED   TARGETREFKIND   TARGETREFNAME   AGE
dnspolicy       True       False      Gateway         external        14m
dnspolicy-t1a   True       False      Gateway         external        14m
dnspolicy-t1b   True       False      Gateway         external        14m

[maleck13]

@maksymvavilov all looks good. I do wonder though about the validity of saying the policy isn't enforced if it has discoverd something unhealthy? Perhaps we should add a new condition Healthy and set that to true if no health checks and published and true or false based on health checks of records?