[Linkerd-proxy] Failed to obtain identity status: InvalidArgument

[mmosbaek]

Hello!

Linkerd v2.14.7

We observed an issue with the Linkerd proxies, that happened for approx. 15-20% of all our Linkerd enabled services within a timespan of around 3 minutes.

[Linkerd-proxy] Failed to obtain identity status: InvalidArgument, message: "[invalid bearer token, service account token has been invalidated]"

We've only seen this issue once or twice, and it seems to resolve it self, but the short outage seem quite peculiar. We can't really find any corresponding events that could have caused this.

Hoping you could give us some pointers on what to look for.

Best regards

[alpeb]

It seems there's an issue with the ServiceAccount token linkerd-identity-token that gets inserted into the pod, as a projected volume. The kubelet is supposed to rotate that token before 80% of its lifetime is attained. Maybe there's a wider issue in your cluster preventing that? If you have access to the kubelet logs in the offending node that might help.

[mmosbaek]

Thank you.

We might have access to the kubelet logs. I will look into it.

[mmosbaek]

unfortunately we do not have access to the kubelet logs on the node. But it does look like all services with this issue was located on the same node. I think we can conclude that something was wrong with that specific node. The node has been rolled, so I don't think we'll get closer to the culprit this time.

Thank you for the hint. If we see the issue again, I will certainly keep an eye on the kubelet logs.