Issue with missing internal IP for nodes created by driver

[Jasstkn]

Hi!

I'm trying to investigate the problem with metrics-server. In the logs I see that it's complaining about missing internal IP:

unable to extract connection information for node "sandbox-infra1": node sandbox-infra1 had no addresses that matched types [InternalIP]]

It's working fine only for node which aren't in the Hetzner Cloud (dedicated root server). From the kubectl get nodes -o wide, I can see that this node automatically got the internal IP and other only external IPs.

NAME                        STATUS   ROLES          AGE     VERSION   INTERNAL-IP      EXTERNAL-IP      OS-IMAGE             KERNEL-VERSION     CONTAINER-RUNTIME
controlplane-1  Ready    controlplane   55m     v1.20.5   <none>           XXXXXXX   Ubuntu 20.04.2 LTS   5.4.0-72-generic   docker://20.10.6
controlplane-2       Ready    etcd           17d     v1.20.5   <none>           XXXXXXX     Ubuntu 20.04.2 LTS   5.4.0-66-generic   docker://20.10.5
data1               Ready    worker         17d     v1.20.5   XXXXXXX          <none>      Ubuntu 20.10         5.8.0-48-generic   docker://20.10.5
infra1              Ready    worker         17d     v1.20.5   <none>           XXXXXXX    Ubuntu 20.04.2 LTS   5.4.0-66-generic   docker://20.10.5
infra2              Ready    worker         36m     v1.20.5   <none>           XXXXXXX     Ubuntu 20.04.2 LTS   5.4.0-72-generic   docker://20.10.6
worker1             Ready    worker         2d13h   v1.20.5   <none>           XXXXXXX    Ubuntu 20.04.2 LTS   5.4.0-72-generic   docker://20.10.6

Do you have any ideas what is going on and how to fix it?
I attached dedicated server using the following command:

sudo docker run -d --privileged \
  --restart=unless-stopped \
  --net=host \
  -v /etc/kubernetes:/etc/kubernetes \
  -v /var/run:/var/run \
  rancher/rancher-agent:v2.5.7 \
  --server https://rancher.example.com \
  --token XXXXXX \
  --node-name data1 \
  --label type=dedicated \
  --taints dedicated=true:NoSchedule

Those nodes are attached to the same private network.

[thomashoell]

I'm having a similar problem. After setting up the cluster, I couldn't get logs from the worker node. Only after creating a firewall rule that allows traffic from the controller node's external IP address to worker:10250, it worked.

All nodes, including the Rancher server, are in the same private network:

Interestingly, kubectl reports the node's external IP als internal while the external IP seems to be missing:

NAME      STATUS   ROLES                      AGE     VERSION    INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION     CONTAINER-RUNTIME
master1   Ready    controlplane,etcd,worker   5d15h   v1.20.11   65.108.xx.xx    <none>        Ubuntu 20.04.3 LTS   5.4.0-84-generic   docker://20.10.7
worker1   Ready    worker                     5d15h   v1.20.11   65.108.xx.xxx   <none>        Ubuntu 20.04.3 LTS   5.4.0-84-generic   cri-o://1.20.5

Rancher seems to get the node's IP addresses correctly though

The issue here seems to be the metrics server as well:

E0925 13:18:00.108492 1 server.go:132] unable to fully scrape metrics: unable to fully scrape metrics from node worker1: unable to fetch metrics from node worker1: Get "https://65.108.xx.xxx:10250/stats/summary?only_cpu_and_memory=true": dial tcp 65.108.xx.xxx:10250: i/o timeout