fix: Default to kubernetes pod IP instead of deprecated pod DNS names

[johanandren]

Naive search replace to see what breaks in CI

References #1242

[vcanuel]

If I may, we are looking for pods (in https://github.com/akka/akka-management/blob/6a49c2c0f29959976d3a6a903db3c7761e8afab1/discovery-kubernetes-api/src/main/scala/akka/discovery/kubernetes/KubernetesApiServiceDiscovery.scala#L149C16-L149C16) but maybe we need to find the service instead. Thanks for looking at this issue :)

[johanandren]

Hmm, then I think I changed too much to start with, the label selection should still be about pods and I'll have to dig a bit deeper than a search-replace.

As far as I understand the only thing we want to change here is to not use the pod dns names as those are deprecated.

[johanandren]

The problem is actually that the svc name does not resolve unless there is a service collecting the pods and the pods are ready. Usually Akka forming cluster is something you want to wait for before marking the pods as ready, to not accept requests before a node has joined the cluster.

I wonder if the right thing isn't just to switch to akka.discovery.kubernetes-api.use-raw-ip=true to avoid using the deprecated dns names for the pods.

[vcanuel]

The problem is actually that the svc name does not resolve unless there is a service collecting the pods and the pods are ready. Usually Akka forming cluster is something you want to wait for before marking the pods as ready, to not accept requests before a node has joined the cluster.

I wonder if the right thing isn't just to switch to akka.discovery.kubernetes-api.use-raw-ip=true to avoid using the deprecated dns names for the pods.

That's how we fixed it indeed ;)

[johanandren]

Thanks for verifying, then that should probably be the default (and should anyway work with older Kubernetes versions as well)

[octonato]

LGTM