External-DNS running into Timeouts (503) getting records from Infoblox

[iskander-el-baraka]

Situation: We've configured an AKS cluster in Azure and Infoblox appliances who are duplicating the dns zones. After installing and configuring external-dns, we are running into CrashLoopBackOff for the pod with the following error:

2024/09/11 08:20:31 GetObject request error: 'WAPI request error: 503('503 Service Temporarily Unavailable')
Contents:
503 Service Temporarily Unavailable
Service Temporarily Unavailable, The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.
cannot unmarshall '', err: 'unexpected end of JSON input'

Enabling the debug mode on the Infoblox appliance shows additional error messages similar to:
2024/09/10 11:51:33 GetObject request error: 'Get "https://HOST:443/wapi/v2.3.1/record:host?\_max_results=500&_proxy_search=GM&_return_fields=
extattrs%2Cipv4addrs%2Cipv6addrs%2Cname%2Cview%2Czone%2Ccomment%2Cnetwork_view%2Caliases%2Cuse_ttl%2Cttl%2Cconfigure_for_dns&zone=ZONE": EOF'

Solution direction:
The External-dns webhook and/or the infoblox-go-client uses proxy_search in some of the code resulting in the API call being redirected to the gridmaster which causes issues when it that cant be reached or isn't used/present.

_proxy_search from the wapi docs: https://ipam.illinois.edu/wapidoc/#objects
If set to ‘GM’, the request is redirected to Grid master for processing. If set to ‘LOCAL’, the request is processed locally. This option is applicable only on vConnector grid members. The default is ‘LOCAL’.

This can be replicated with a simple curl command or invoke-restmethod:
https://HOST:443/wapi/v2.3.1/record:host?_max_results=500 -> works
https://HOST:443/wapi/v2.3.1/record:host?_max_results=500&_proxy_search=GM -> doesnt work

References:
This function is crashing we think:

external-dns-infoblox-webhook/internal/infoblox/infoblox.go

Line 190 in 3fddeb4

func (p *Provider) Records(_ context.Context) (endpoints []*endpoint.Endpoint, err error) {

Which is reliant on ibclient.WapiRequestBuilder found here: https://github.com/infobloxopen/infoblox-go-client/blob/master/connector.go