connect sidecar gets "unauthenticated: ACL not found" while replacing node in AKS cluster

[MageshSrinivasulu]

I am facing a problem when the pod starts up the envoy sidecar fails with the error. This happens while performing maintenance over AKS nodes where I need to move the connect inject enable pods to newer nodes.

[2023-10-03 05:44:33.997][1][warning][config] [./source/common/config/grpc_stream.h:160] DeltaAggregatedResources gRPC config stream closed: 16, unauthenticated: ACL not found

A similar issue is being discussed in Nomad as well hashicorp/nomad#9785

[Ranjandas]

Hi @MageshSrinivasulu, Could you please provide more information about your setup?

• What version of Consul-K8S/Helm Chart are you using?
• Are you consistently able to reproduce this issue? Do you have any sequence of instructions to reproduce this?
• Is this issue happening on a specific kind of workload (StatefulSet vs Deployments/Pods etc)?

[MageshSrinivasulu]

Hi @Ranjandas

1. Consul helm 0.49.5
2. Yes able to consistently reproduce this issue. This happens every time we do maintenance on Kubernetes nodes like node replacement or new node pool creation
3. No it is happening at random across the deployment / stateful set that is using consul connect inject