rpc error: code = Unavailable desc = last connection error: connection error: desc = "transport: authentication handshake failed: tls.... #9704

talosnoob · 2024-11-12T08:14:04Z

talosnoob
Nov 12, 2024

Hello Talos team

It seems I made a newbie mistake and now I don't know how to fix it :(

I had a working cluster. At some point I decided to do "talosctl gen config" with the --force option for it and as a result I got 3 new files:

controlplane.yaml, worker.yaml, talosconfig

which erased similar files that was before executing the command with the --force key.

Then I copied the talosconfig file with replacement to the ~/.talos/config directory.

Now when I try to access my cluster using the talosctl utility, I get a certificate error:

rpc error: code = Unavailable desc = last connection error: connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of "x509: Ed25519 verification failure" while trying to verify candidate authority certificate "talos")"

Is there a way to fix this and restore access to my cluster? Or I must deploy it "from scratch" again ?

I'll saw the very similar discussion in this forum (#3704) but there is no answer for user QuinnBast question and in my case I already have working kubernetes cluster but can't access to it with talosctl utility.

# kubectl get nodes -o wide
NAME       STATUS   ROLES           AGE   VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE         KERNEL-VERSION   CONTAINER-RUNTIME
cp-node1   Ready    control-plane   37d   v1.31.1   192.168.50.11   <none>        Talos (v1.8.0)   6.6.52-talos     containerd://2.0.0-rc.4
cp-node2   Ready    control-plane   37d   v1.31.1   192.168.50.12   <none>        Talos (v1.8.0)   6.6.52-talos     containerd://2.0.0-rc.4
wr-node1   Ready    <none>          37d   v1.31.1   192.168.50.21   <none>        Talos (v1.8.0)   6.6.52-talos     containerd://2.0.0-rc.4
wr-node2   Ready    <none>          36d   v1.31.1   192.168.50.22   <none>        Talos (v1.8.0)   6.6.52-talos     containerd://2.0.0-rc.4
wr-node3   Ready    <none>          36d   v1.31.1   192.168.50.23   <none>        Talos (v1.8.0)   6.6.52-talos     containerd://2.0.0-rc.4

# kubectl get services
NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
kubernetes   ClusterIP   10.96.0.1        <none>        443/TCP        37d
nginx-demo   NodePort    10.103.114.251   <none>        80:30643/TCP   6d13h

# curl http://192.168.50.21:30643
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

# talosctl get members
rpc error: code = Unavailable desc = last connection error: connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"x509: Ed25519 verification failure\" while trying to verify candidate authority certificate \"talos\")"

Answered by smira

Nov 12, 2024

As you create new client-side talosconfig file, you erased the previous client-side configuration including the cluster secrets, etc.

So unless you have a backup, the access to the cluster is lost.

You could do a little hack though (which works today, but might not work in the future): as you have admin-level kubectl access, you can create a privileged pod on the controlplane node with something like:

kubectl debug -it node/cp-node-1 --image alpine --profile=sysadmin -n kube-system

From within that pod, you can copy out the file /host/system/state/config.yaml, which will be your controlplane.yaml file (from gen config). After that you can follow the doc.

Note: you seem to have 2 controlp…

View full answer

smira · 2024-11-12T09:37:22Z

smira
Nov 12, 2024
Maintainer

As you create new client-side talosconfig file, you erased the previous client-side configuration including the cluster secrets, etc.

So unless you have a backup, the access to the cluster is lost.

You could do a little hack though (which works today, but might not work in the future): as you have admin-level kubectl access, you can create a privileged pod on the controlplane node with something like:

kubectl debug -it node/cp-node-1 --image alpine --profile=sysadmin -n kube-system

From within that pod, you can copy out the file /host/system/state/config.yaml, which will be your controlplane.yaml file (from gen config). After that you can follow the doc.

Note: you seem to have 2 controlplane nodes, which is not recommended from etcd quorum perspective. Do either one or three.

1 reply

talosnoob Nov 12, 2024
Author

Thank's Andrey.
Your recommendation was really helpfull.
Now I know - it's better to make a backup copy of my talosconfig.

# talosctl health -n 192.168.50.11
discovered nodes: ["192.168.50.11" "192.168.50.12" "192.168.50.21" "192.168.50.22" "192.168.50.23"]
waiting for etcd to be healthy: ...
waiting for etcd to be healthy: OK
waiting for etcd members to be consistent across nodes: ...
waiting for etcd members to be consistent across nodes: OK
waiting for etcd members to be control plane nodes: ...
waiting for etcd members to be control plane nodes: OK
waiting for apid to be ready: ...
waiting for apid to be ready: OK
waiting for all nodes memory sizes: ...
waiting for all nodes memory sizes: OK
waiting for all nodes disk sizes: ...
waiting for all nodes disk sizes: OK
waiting for no diagnostics: ...
waiting for no diagnostics: OK
waiting for kubelet to be healthy: ...
waiting for kubelet to be healthy: OK
waiting for all nodes to finish boot sequence: ...
waiting for all nodes to finish boot sequence: OK
waiting for all k8s nodes to report: ...
waiting for all k8s nodes to report: OK
waiting for all control plane static pods to be running: ...
waiting for all control plane static pods to be running: OK
waiting for all control plane components to be ready: ...
waiting for all control plane components to be ready: OK
waiting for all k8s nodes to report ready: ...
waiting for all k8s nodes to report ready: OK
waiting for kube-proxy to report ready: ...
waiting for kube-proxy to report ready: OK
waiting for coredns to report ready: ...
waiting for coredns to report ready: OK
waiting for all k8s nodes to report schedulable: ...
waiting for all k8s nodes to report schedulable: OK

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rpc error: code = Unavailable desc = last connection error: connection error: desc = "transport: authentication handshake failed: tls.... #9704

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 1 comment 1 reply

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

rpc error: code = Unavailable desc = last connection error: connection error: desc = "transport: authentication handshake failed: tls.... #9704

talosnoob Nov 12, 2024

Replies: 1 comment · 1 reply

smira Nov 12, 2024 Maintainer

talosnoob Nov 12, 2024 Author

talosnoob
Nov 12, 2024

Replies: 1 comment 1 reply

smira
Nov 12, 2024
Maintainer

talosnoob Nov 12, 2024
Author