Fleet seems unable to communicate within itself ( Get \"https://10.43.140.17/k8s/clusters/local/version?timeout=15s\": dial tcp 10.43.140.17:443: connect: no route to host, requeuing")

[baptisterajaut]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

All my cluster are showing an error in upstream cluster, eventough the local and production work
And i cant seem to import a new cluster nor recover one.
I am absolutely clueless on how to fix that.

Expected Behavior

I should be able to import my clusters and create new ones.

Steps To Reproduce

Not sure. I had rancher 2.8.5 and i updated to 2.9.1.

Environment

- Architecture: amd64
- Rancher 2.9.1 - Single docker node
- Fleet Version: v0.10.1 
- Cluster:
  - Provider: rke2
  - Options: none
  - Kubernetes Version: 1.28.10+rke2

Logs

time="2024-09-13T08:45:00Z" level=error msg="error syncing 'fleet-default/production': handler import-cluster: Get \"https://10.43.140.17/k8s/clusters/c-m-rh7vjf7x/version?timeout=15s\": dial tcp 10.43.140.17:443: connect: no route to host, requeuing"
time="2024-09-13T08:46:15Z" level=error msg="Cluster import for 'fleet-default/bureau'. Smoke test failed: Get \"https://10.43.140.17/k8s/clusters/c-m-p7k54glf/version?timeout=15s\": dial tcp 10.43.140.17:443: connect: no route to host"
time="2024-09-13T08:46:15Z" level=error msg="error syncing 'fleet-default/bureau': handler import-cluster: Get \"https://10.43.140.17/k8s/clusters/c-m-p7k54glf/version?timeout=15s\": dial tcp 10.43.140.17:443: connect: no route to host, requeuing"
time="2024-09-13T08:46:21Z" level=error msg="Cluster import for 'fleet-default/bureau'. Smoke test failed: Get \"https://10.43.140.17/k8s/clusters/c-m-p7k54glf/version?timeout=15s\": dial tcp 10.43.140.17:443: connect: no route to host"
time="2024-09-13T08:46:21Z" level=error msg="error syncing 'fleet-default/bureau': handler import-cluster: Get \"https://10.43.140.17/k8s/clusters/c-m-p7k54glf/version?timeout=15s\": dial tcp 10.43.140.17:443: connect: no route to host, requeuing"
time="2024-09-13T08:47:20Z" level=error msg="Cluster import for 'fleet-default/testcluster'. Smoke test failed: Get \"https://10.43.140.17/k8s/clusters/c-m-f9qr9b5w/version?timeout=15s\": dial tcp 10.43.140.17:443: connect: no route to host"
time="2024-09-13T08:47:20Z" level=error msg="error syncing 'fleet-default/testcluster': handler import-cluster: Get \"https://10.43.140.17/k8s/clusters/c-m-f9qr9b5w/version?timeout=15s\": dial tcp 10.43.140.17:443: connect: no route to host, requeuing"
time="2024-09-13T08:51:27Z" level=error msg="Cluster import for 'fleet-default/production'. Smoke test failed: Get \"https://10.43.140.17/k8s/clusters/c-m-rh7vjf7x/version?timeout=15s\": dial tcp 10.43.140.17:443: connect: no route to host"
time="2024-09-13T08:51:27Z" level=error msg="error syncing 'fleet-default/production': handler import-cluster: Get \"https://10.43.140.17/k8s/clusters/c-m-rh7vjf7x/version?timeout=15s\": dial tcp 10.43.140.17:443: connect: no route to host, requeuing"
time="2024-09-13T08:51:46Z" level=error msg="Cluster import for 'fleet-local/local'. Smoke test failed: Get \"https://10.43.140.17/k8s/clusters/local/version?timeout=15s\": dial tcp 10.43.140.17:443: connect: no route to host"
time="2024-09-13T08:51:46Z" level=error msg="error syncing 'fleet-local/local': handler import-cluster: Get \"https://10.43.140.17/k8s/clusters/local/version?timeout=15s\": dial tcp 10.43.140.17:443: connect: no route to host, requeuing"

Anything else?

[mmartin24]

Hello @baptisterajaut.
I tried to reproduce this issue using single node k3s + separate downstream cluster with gitrepos both locally and on default and upgrading from 2.8.5 to 2.9.1and worked ok.

Could you please provide further steps on how the original set up was deployed?

[baptisterajaut]

Thank you @mmartin24 so much for your quick reply. This was actually not on fleet part. I spent the entire week trying to find the underlying problem, and found it. Its on rancher part rancher/rancher#47102

Sorry for inconvenience.

[mmartin24]

Thank you @mmartin24 so much for your quick reply. This was actually not on fleet part. I spent the entire week trying to find the underlying problem, and found it. Its on rancher part rancher/rancher#47102

Sorry for inconvenience.

Not at all @baptisterajaut. Thanks to you for reaching out.