Linkerd proxy-to-proxy container calls failing with reason "Invalid Peer Certificate CertExpired" despite root and issuer cert unexpired. #11581

bhanuprakash-1 · 2023-11-06T18:42:22Z

bhanuprakash-1
Nov 6, 2023

My AKS Cluster suddenly started to fail all the traffic and upon checking the Proxy Container Logs, it shows that invalid peer certificate contents: invalid peer certificate: CertExpired error.

Traffic between dataplane proxy container (pod to pod traffic) and also traffic between proxy containers and linkerd control plane pods like identity:controller is broken, with similar error related to **Sending fatal alert BadCertificate ** and invalid peer certificate: CertExpired.

Also, no new pods are coming up because of linkerd-proxy container failed to become ready within 120s timeout. These new proxy containers also has the same issue of invalid peer certificate: CertExpired while Waiting for identity to be initialized.

I have verified my linkerd Trust Anchor certification ( also called root certificate) and issuer certificate and both are not expired.
So not use what is the issue.

The LINKERD2_PROXY_IDENTITY_TRUST_ANCHORS certificate is same for all the dataplane linkerd-proxy containers and also same for the control plane linkerd-identity pods.

We use linkerd only for mTLS functionality. We don't use linkerd for routing, logging, retrying etc.

Attaching some logs below:

linkerd-proxy container startup failure:

Containers:
  linkerd-proxy:
    Container ID:  containerd://8bf9c783e384b90ee70f6fde28b7b33b347b50fd6070399adb29e3a4389ad250
    Image:         mcr.microsoft.com/oss/linkerd/proxy:2.13.1
    Image ID:      mcr.microsoft.com/oss/linkerd/proxy@sha256:9a15d0ddafbaf912fadf2008a9db475bead37cfcf404f78d7617f85e8cd4c04a
    Ports:         4143/TCP, 4191/TCP
    Host Ports:    0/TCP, 0/TCP
    State:         Waiting
      Reason:      CrashLoopBackOff
    Last State:    Terminated
      Reason:      Error
      Message:     nect error=endpoint 172.18.33.197:8090: invalid peer certificate contents: invalid peer certificate: CertExpired error.sources=[invalid peer certificate contents: invalid peer certificate: CertExpired]
[   149.638311s]  WARN ThreadId(01) watch{port=80}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}:endpoint{addr=172.18.33.144:8090}: rustls::conn: Sending fatal alert BadCertificate
[   149.638389s]  WARN ThreadId(01) watch{port=80}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}:endpoint{addr=172.18.33.144:8090}: linkerd_reconnect: Failed to connect error=endpoint 172.18.33.144:8090: invalid peer certificate contents: invalid peer certificate: CertExpired error.sources=[invalid peer certificate contents: invalid peer certificate: CertExpired]
[   149.680366s]  WARN ThreadId(01) watch{port=80}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}:endpoint{addr=172.18.34.189:8090}: rustls::conn: Sending fatal alert BadCertificate
[   149.680453s]  WARN ThreadId(01) watch{port=80}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}:endpoint{addr=172.18.34.189:8090}: linkerd_reconnect: Failed to connect error=endpoint 172.18.34.189:8090: invalid peer certificate contents: invalid peer certificate: CertExpired error.sources=[invalid peer certificate contents: invalid peer certificate: CertExpired]
[   149.680595s]  WARN ThreadId(01) watch{port=80}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}:endpoint{addr=172.18.33.197:8090}: rustls::conn: Sending fatal alert BadCertificate
[   149.680661s]  WARN ThreadId(01) watch{port=80}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}:endpoint{addr=172.18.33.197:8090}: linkerd_reconnect: Failed to connect error=endpoint 172.18.33.197:8090: invalid peer certificate contents: invalid peer certificate: CertExpired error.sources=[invalid peer certificate contents: invalid peer certificate: CertExpired]
[   150.013080s]  WARN ThreadId(01) linkerd_app: Waiting for identity to be initialized...

      Exit Code:    137
      Started:      Mon, 06 Nov 2023 09:06:56 -0800
      Finished:     Mon, 06 Nov 2023 09:09:26 -0800
    Ready:          False
    Restart Count:  77
    Limits:
      cpu:     1
      memory:  500Mi
    Requests:
      cpu:      300m
      memory:   40Mi
    Liveness:   http-get http://:4191/live delay=10s timeout=1s period=10s #success=1 #failure=3
    Readiness:  http-get http://:4191/ready delay=2s timeout=1s period=10s #success=1 #failure=3
    Environment:
      _pod_name:                                                data-plane--farm-service-56f485965d-x8nkb (v1:metadata.name)
      _pod_ns:                                                  default (v1:metadata.namespace)
      _pod_nodeName:                                             (v1:spec.nodeName)
      LINKERD2_PROXY_LOG:                                       warn,linkerd=info
      LINKERD2_PROXY_LOG_FORMAT:                                plain
      LINKERD2_PROXY_DESTINATION_SVC_ADDR:                      linkerd-dst-headless.linkerd.svc.cluster.local.:8086
      LINKERD2_PROXY_DESTINATION_PROFILE_NETWORKS:              10.0.0.0/8,100.64.0.0/10,172.16.0.0/12,192.168.0.0/16
      LINKERD2_PROXY_POLICY_SVC_ADDR:                           linkerd-policy.linkerd.svc.cluster.local.:8090
      LINKERD2_PROXY_POLICY_WORKLOAD:                           $(_pod_ns):$(_pod_name)
      LINKERD2_PROXY_INBOUND_DEFAULT_POLICY:                    all-unauthenticated
      LINKERD2_PROXY_POLICY_CLUSTER_NETWORKS:                   10.0.0.0/8,100.64.0.0/10,172.16.0.0/12,192.168.0.0/16
      LINKERD2_PROXY_INBOUND_CONNECT_TIMEOUT:                   100ms
      LINKERD2_PROXY_OUTBOUND_CONNECT_TIMEOUT:                  10s
      LINKERD2_PROXY_CONTROL_LISTEN_ADDR:                       0.0.0.0:4190
      LINKERD2_PROXY_ADMIN_LISTEN_ADDR:                         0.0.0.0:4191
      LINKERD2_PROXY_OUTBOUND_LISTEN_ADDR:                      127.0.0.1:4140
      LINKERD2_PROXY_INBOUND_LISTEN_ADDR:                       0.0.0.0:4143
      LINKERD2_PROXY_INBOUND_IPS:                                (v1:status.podIPs)
      LINKERD2_PROXY_INBOUND_PORTS:                             80
      LINKERD2_PROXY_DESTINATION_PROFILE_SUFFIXES:              svc.cluster.local.
      LINKERD2_PROXY_INBOUND_ACCEPT_KEEPALIVE:                  10000ms
      LINKERD2_PROXY_OUTBOUND_CONNECT_KEEPALIVE:                10000ms
      LINKERD2_PROXY_INBOUND_PORTS_DISABLE_PROTOCOL_DETECTION:  25,587,3306,4444,5432,6379,9300,11211
      LINKERD2_PROXY_DESTINATION_CONTEXT:                       {"ns":"$(_pod_ns)", "nodeName":"$(_pod_nodeName)"}

      _pod_sa:                                                   (v1:spec.serviceAccountName)
      _l5d_ns:                                                  linkerd
      _l5d_trustdomain:                                         cluster.local
      LINKERD2_PROXY_IDENTITY_DIR:                              /var/run/linkerd/identity/end-entity
      LINKERD2_PROXY_IDENTITY_TRUST_ANCHORS:                   <Cert here, which is not expired>

      LINKERD2_PROXY_IDENTITY_TOKEN_FILE:                       /var/run/secrets/tokens/linkerd-identity-token
      LINKERD2_PROXY_IDENTITY_SVC_ADDR:                         linkerd-identity-headless.linkerd.svc.cluster.local.:8080
      LINKERD2_PROXY_IDENTITY_LOCAL_NAME:                       $(_pod_sa).$(_pod_ns).serviceaccount.identity.linkerd.cluster.local
      LINKERD2_PROXY_IDENTITY_SVC_NAME:                         linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local
      LINKERD2_PROXY_DESTINATION_SVC_NAME:                      linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local
      LINKERD2_PROXY_POLICY_SVC_NAME:                           linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local
    Mounts:
      /var/run/linkerd/identity/end-entity from linkerd-identity-end-entity (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-d8bkf (ro)
      /var/run/secrets/tokens from linkerd-identity-token (rw)

Already running Linkerd-proxy logs

[233615.263893s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:endpoint{addr=172.18.33.222:8080}: linkerd_reconnect: Failed to connect error=endpoint 172.18.33.222:8080: invalid peer certificate contents: invalid peer certificate: CertExpired error.sources=[invalid peer certificate contents: invalid peer certificate: CertExpired]
[233615.270531s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:endpoint{addr=172.18.34.193:8080}: rustls::conn: Sending fatal alert BadCertificate    
[233615.270591s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:endpoint{addr=172.18.34.193:8080}: linkerd_reconnect: Failed to connect error=endpoint 172.18.34.193:8080: invalid peer certificate contents: invalid peer certificate: CertExpired error.sources=[invalid peer certificate contents: invalid peer certificate: CertExpired]
[233615.750697s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:endpoint{addr=172.18.34.50:8080}: rustls::conn: Sending fatal alert BadCertificate    
[233615.750755s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:endpoint{addr=172.18.34.50:8080}: linkerd_reconnect: Failed to connect error=endpoint 172.18.34.50:8080: invalid peer certificate contents: invalid peer certificate: CertExpired error.sources=[invalid peer certificate contents: invalid peer certificate: CertExpired]
[233615.766762s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:endpoint{addr=172.18.33.222:8080}: rustls::conn: Sending fatal alert BadCertificate    
[233615.766816s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:endpoint{addr=172.18.33.222:8080}: linkerd_reconnect: Failed to connect error=endpoint 172.18.33.222:8080: invalid peer certificate contents: invalid peer certificate: CertExpired error.sources=[invalid peer certificate contents: invalid peer certificate: CertExpired]
[233615.772385s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:endpoint{addr=172.18.34.193:8080}: rustls::conn: Sending fatal alert BadCertificate    
[233615.772434s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:endpoint{addr=172.18.34.193:8080}: linkerd_reconnect: Failed to connect error=endpoint 172.18.34.193:8080: invalid peer certificate contents: invalid peer certificate: CertExpired error.sources=[invalid peer certificate contents: invalid peer certificate: CertExpired]
[233616.254167s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:endpoint{addr=172.18.34.50:8080}: rustls::conn: Sending fatal alert BadCertificate    
[233616.254259s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:endpoint{addr=172.18.34.50:8080}: linkerd_reconnect: Failed to connect error=endpoint 172.18.34.50:8080: invalid peer certificate contents: invalid peer certificate: CertExpired error.sources=[invalid peer certificate contents: invalid peer certificate: CertExpired]
[233616.270237s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:endpoint{addr=172.18.33.222:8080}: rustls::conn: Sending fatal alert BadCertificate    
[233616.270298s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:endpoint{addr=172.18.33.222:8080}: linkerd_reconnect: Failed to connect error=endpoint 172.18.33.222:8080: invalid peer certificate contents: invalid peer certificate: CertExpired error.sources=[invalid peer certificate contents: invalid peer certificate: CertExpired]
[233616.275531s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:endpoint{addr=172.18.34.193:8080}: rustls::conn: Sending fatal alert BadCertificate    
[233616.275582s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:endpoint{addr=172.18.34.193:8080}: linkerd_reconnect: Failed to connect error=endpoint 172.18.34.193:8080: invalid peer certificate contents: invalid peer certificate: CertExpired error.sources=[invalid peer certificate contents: invalid peer certificate: CertExpired]
[233616.759854s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:endpoint{addr=172.18.34.50:8080}: rustls::conn: Sending fatal alert BadCertificate    
[233616.759917s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:endpoint{addr=172.18.34.50:8080}: linkerd_reconnect: Failed to connect error=endpoint 172.18.34.50:8080: invalid peer certificate contents: invalid peer certificate: CertExpired error.sources=[invalid peer certificate contents: invalid peer certificate: CertExpired]
[233616.772546s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:endpoint{addr=172.18.33.222:8080}: rustls::conn: Sending fatal alert BadCertificate    
[233616.772601s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:endpoint{addr=172.18.33.222:8080}: linkerd_reconnect: Failed to connect error=endpoint 172.18.33.222:8080: invalid peer certificate contents: invalid peer certificate: CertExpired error.sources=[invalid peer certificate contents: invalid peer certificate: CertExpired]
[233616.778463s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:endpoint{addr=172.18.34.193:8080}: rustls::conn: Sending fatal alert BadCertificate    
[233616.778507s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:endpoint{addr=172.18.34.193:8080}: linkerd_reconnect: Failed to connect error=endpoint 172.18.34.193:8080: invalid peer certificate contents: invalid peer certificate: CertExpired error.sources=[invalid peer certificate contents: invalid peer certificate: CertExpired]
[233617.000887s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_stack::failfast: Service entering failfast after 3s
[233617.000960s] ERROR ThreadId(02) identity: linkerd_proxy_identity_client::certify: Failed to obtain identity error=status: Unknown, message: "controller linkerd-identity-headless.linkerd.svc.cluster.local:8080: service in fail-fast", details: [], metadata: MetadataMap { headers: {} } error.sources=[controller linkerd-identity-headless.linkerd.svc.cluster.local:8080: service in fail-fast, service in fail-fast]
[233618.159913s] ERROR ThreadId(01) inbound:server{port=80}: rustls::conn: TLS alert received: AlertMessagePayload {
    level: Fatal,
    description: BadCertificate,
}    
[233618.266110s] ERROR ThreadId(01) inbound:server{port=80}: rustls::conn: TLS alert received: AlertMessagePayload {
    level: Fatal,
    description: BadCertificate,
}

Linkerd check --proxy output.
Although this says that CRD(s) are missing, when I run this command in other healthy clusters, the result is same i.e, CRDs missing. Not sure what is the issue, but seems not related to the current issue.

bhanu@Azure:~$ linkerd check --proxy
kubernetes-api
--------------
√ can initialize the client
√ can query the Kubernetes API
 
kubernetes-version
------------------
√ is running the minimum Kubernetes API version
 
linkerd-existence
-----------------
√ 'linkerd-config' config map exists
√ heartbeat ServiceAccount exist
√ control plane replica sets are ready
√ no unschedulable pods
√ control plane pods are ready
√ cluster networks contains all pods
√ cluster networks contains all services
 
linkerd-config
--------------
√ control plane Namespace exists
√ control plane ClusterRoles exist
√ control plane ClusterRoleBindings exist
√ control plane ServiceAccounts exist
× control plane CustomResourceDefinitions exist
    CRD httproutes.policy.linkerd.io is missing version v1beta3, missing httproutes.gateway.networking.k8s.io
    see https://linkerd.io/2.14/checks/#l5d-existence-crd for hints
 
Status check results are ×

mateiidavid · 2023-11-07T10:35:32Z

mateiidavid
Nov 7, 2023

What does linkerd check output? You've only included the proxy check. Without specifying --proxy we should get a fuller view of the install configuration.

2 replies

bhanuprakash-1 Nov 7, 2023
Author

Hi @mateiidavid , thanks for the reply.

The output is same for both linkerd check and 'linkerd check --proxy`.


bhanu@Azure:~$ linkerd check 
kubernetes-api
--------------
√ can initialize the client
√ can query the Kubernetes API
 
kubernetes-version
------------------
√ is running the minimum Kubernetes API version
 
linkerd-existence
-----------------
√ 'linkerd-config' config map exists
√ heartbeat ServiceAccount exist
√ control plane replica sets are ready
√ no unschedulable pods
√ control plane pods are ready
√ cluster networks contains all pods
√ cluster networks contains all services
 
linkerd-config
--------------
√ control plane Namespace exists
√ control plane ClusterRoles exist
√ control plane ClusterRoleBindings exist
√ control plane ServiceAccounts exist
× control plane CustomResourceDefinitions exist
    CRD httproutes.policy.linkerd.io is missing version v1beta3, missing httproutes.gateway.networking.k8s.io
    see https://linkerd.io/2.14/checks/#l5d-existence-crd for hints
 
Status check results are ×
bhanu@Azure:~$
has context menu

bhanuprakash-1 Nov 7, 2023
Author

Please reply if you need more information. Thanks!

bhanuprakash-1 · 2023-11-07T11:54:26Z

bhanuprakash-1
Nov 7, 2023
Author

Also adding the list of CRDs:

bhanu@Azure:~$ kubectl get customResourceDefinitions -n linkerd
NAME                                                             CREATED AT
authorizationpolicies.policy.linkerd.io                          2023-04-30T18:10:21Z
azureassignedidentities.aadpodidentity.k8s.io                    2022-08-25T13:51:33Z
azureidentities.aadpodidentity.k8s.io                            2022-08-25T13:51:33Z
azureidentitybindings.aadpodidentity.k8s.io                      2022-08-25T13:51:33Z
azurepodidentityexceptions.aadpodidentity.k8s.io                 2022-08-25T13:51:33Z
configs.config.gatekeeper.sh                                     2022-08-25T13:34:36Z
constraintpodstatuses.status.gatekeeper.sh                       2022-08-25T13:34:36Z
constrainttemplatepodstatuses.status.gatekeeper.sh               2022-08-25T13:34:36Z
constrainttemplates.templates.gatekeeper.sh                      2022-08-25T13:34:36Z
expansiontemplate.expansion.gatekeeper.sh                        2023-11-07T03:17:26Z
expansiontemplatepodstatuses.status.gatekeeper.sh                2023-11-07T03:17:26Z
httproutes.policy.linkerd.io                                     2023-04-30T18:10:21Z
imagejobs.eraser.sh                                              2023-10-13T05:47:09Z
imagelists.eraser.sh                                             2023-10-13T05:47:10Z
k8sazurecustomcontainerallowedimages.constraints.gatekeeper.sh   2023-03-16T23:46:29Z
k8sazurev1containerallowedports.constraints.gatekeeper.sh        2022-09-14T18:20:41Z
k8sazurev1externalips.constraints.gatekeeper.sh                  2022-09-14T18:20:41Z
k8sazurev1ingresshttpsonly.constraints.gatekeeper.sh             2022-09-14T17:50:52Z
k8sazurev1serviceallowedports.constraints.gatekeeper.sh          2022-09-14T18:35:41Z
k8sazurev2blockhostnamespace.constraints.gatekeeper.sh           2022-09-14T17:50:42Z
k8sazurev2containerallowedimages.constraints.gatekeeper.sh       2022-09-14T17:50:41Z
k8sazurev2noprivilege.constraints.gatekeeper.sh                  2022-09-14T17:50:41Z
k8sazurev2volumetypes.constraints.gatekeeper.sh                  2022-09-14T18:20:41Z
k8sazurev3allowedcapabilities.constraints.gatekeeper.sh          2022-09-14T17:50:45Z
k8sazurev3allowedseccomp.constraints.gatekeeper.sh               2022-09-14T18:20:45Z
k8sazurev3allowedusersgroups.constraints.gatekeeper.sh           2022-09-14T18:35:41Z
k8sazurev3containerlimits.constraints.gatekeeper.sh              2022-09-14T17:50:41Z
k8sazurev3hostfilesystem.constraints.gatekeeper.sh               2022-09-14T18:50:41Z
k8sazurev3hostnetworkingports.constraints.gatekeeper.sh          2022-09-14T17:50:50Z
k8sazurev3noprivilegeescalation.constraints.gatekeeper.sh        2022-09-14T17:50:47Z
k8sazurev3readonlyrootfilesystem.constraints.gatekeeper.sh       2022-09-14T18:20:42Z
meshtlsauthentications.policy.linkerd.io                         2023-04-30T18:10:21Z
networkauthentications.policy.linkerd.io                         2023-04-30T18:10:21Z
secretproviderclasses.secrets-store.csi.x-k8s.io                 2022-08-25T13:53:14Z
secretproviderclasspodstatuses.secrets-store.csi.x-k8s.io        2022-08-25T13:53:14Z
serverauthorizations.policy.linkerd.io                           2023-04-30T18:10:21Z
servers.policy.linkerd.io                                        2023-04-30T18:10:21Z
serviceprofiles.linkerd.io                                       2023-04-30T18:10:21Z
volumesnapshotclasses.snapshot.storage.k8s.io                    2022-08-25T13:34:17Z
volumesnapshotcontents.snapshot.storage.k8s.io                   2022-08-25T13:34:17Z
volumesnapshots.snapshot.storage.k8s.io                          2022-08-25T13:34:17Z
bhanu@Azure:~$

Also, the linkerd control plane pods are all running fine:

bhanu@Azure:~$ kubectl get pods -n linkerd
NAME                                     READY   STATUS    RESTARTS   AGE
linkerd-destination-5f8cdc54d9-c9l86     4/4     Running   0          3d11h
linkerd-destination-5f8cdc54d9-pxt9p     4/4     Running   0          3d11h
linkerd-destination-5f8cdc54d9-sh2vb     4/4     Running   0          3d11h
linkerd-identity-c6c7f8cc5-4pm2j         2/2     Running   0          3d11h
linkerd-identity-c6c7f8cc5-7gfd4         2/2     Running   0          3d11h
linkerd-identity-c6c7f8cc5-xcj5f         2/2     Running   0          3d12h
linkerd-proxy-injector-58797bdd9-9b64g   2/2     Running   0          3d11h
linkerd-proxy-injector-58797bdd9-qdxkr   2/2     Running   0          3d11h
linkerd-proxy-injector-58797bdd9-snrb8   2/2     Running   0          3d11h
bhanu@Azure:~$

4 replies

mateiidavid Nov 7, 2023

And what linkerd version are you running? Have you upgraded recently and forgot to also upgrade the CRDs? linkerd upgrade --crds. My guess is that the check obfuscates any certificate information because it errors out on the CRD version. I'd suggest fixing this first and then re-running the check command.

You can also print the certificate public information and check the expiration date unless you've already done so.

bhanuprakash-1 Nov 7, 2023
Author

We are running linkerd 2.13.1. We have upgraded to it in April 2023 and have not done any upgrades since then. And also did not face any issues until recently.

I prefer not to run manual commands like linkerd upgrade --crds as we are using linkerd helm charts to install linkerd on our Cluster. I can do it as a one time step if there is no way to debug this.

For linkerd-crds helm chart version : ** 1.6.0**
For linkerd-control-plane helm chart version: 1.12.1

dependencies:

# Get the chart versions list using the command: helm search repo linkerd/linkerd-control-plane --versions
  - name: linkerd-crds
    version: 1.6.0
    repository: https://helm.linkerd.io/stable
    condition: linkerd-crds.enabled
  
  - name: linkerd-control-plane
    version: 1.12.1
    repository: https://helm.linkerd.io/stable
    condition: linkerd-control-plane.enabled

Regarding the certificate, I have checked the expiration date of both root certificate (expiry in 2033) and issuer certificate ( expiry in March 2024).

sharmakhushboo Nov 13, 2023

Hi. I have recently started seeing similar issues in my Azure Kubernetes cluster as well.
Linkerd Version: 2.13.4

In linkerd-proxy logs, I see BadCertificate error(shown below) and 'invalid peer cert' error. Both root and intermediate certificates are valid. So, I think this has to do with leaf certificates that linkerd generated internally for mtls.

The way I mitigated this issue is by first restarting the Linkerd control plane and then restarting the data plane.
If I just restart the data plane, all services go to CrashLoopback error state. So, Linkerd control plan restart is a must. But is not a permanent fix as this issue comes back after some time. (Once every day)
The traffic to the cluster has not increased and I have not made any changes to linkerd setup. Could someone help investigate this issue?

bhanuprakash-1 Nov 14, 2023
Author

Hi @mateiidavid

I have re-installed(helm upgrade --install ) the linkerd helm charts again and that fixed the issue for now ( basically what @sharmakhushboo did)

But as you asked to do linkerd upgrade --crds, I ran the command and then ran the command linker check. Even then the response of linkerd check is same as above i.e, stopping at control plane CustomResourceDefinitions exist stage with same error message.

What else can I do? Appreciate the help. If this happens in production cluster, our service will have an outage :( !!

output of the command linkerd upgrade --crds in the below comment:

Also output of kubectl get crds -A:

bhanu@Azure:~$ kubectl get crds -A
NAME                                                             CREATED AT
authorizationpolicies.policy.linkerd.io                          2023-11-10T15:39:04Z
azureassignedidentities.aadpodidentity.k8s.io                    2022-08-25T13:51:33Z
azureidentities.aadpodidentity.k8s.io                            2022-08-25T13:51:33Z
azureidentitybindings.aadpodidentity.k8s.io                      2022-08-25T13:51:33Z
azurepodidentityexceptions.aadpodidentity.k8s.io                 2022-08-25T13:51:33Z
configs.config.gatekeeper.sh                                     2022-08-25T13:34:36Z
constraintpodstatuses.status.gatekeeper.sh                       2022-08-25T13:34:36Z
constrainttemplatepodstatuses.status.gatekeeper.sh               2022-08-25T13:34:36Z
constrainttemplates.templates.gatekeeper.sh                      2022-08-25T13:34:36Z
expansiontemplate.expansion.gatekeeper.sh                        2023-11-07T03:17:26Z
expansiontemplatepodstatuses.status.gatekeeper.sh                2023-11-07T03:17:26Z
httproutes.policy.linkerd.io                                     2023-11-10T15:39:04Z
imagejobs.eraser.sh                                              2023-10-13T05:47:09Z
imagelists.eraser.sh                                             2023-10-13T05:47:10Z
k8sazurecustomcontainerallowedimages.constraints.gatekeeper.sh   2023-03-16T23:46:29Z
k8sazurev1containerallowedports.constraints.gatekeeper.sh        2022-09-14T18:20:41Z
k8sazurev1externalips.constraints.gatekeeper.sh                  2022-09-14T18:20:41Z
k8sazurev1ingresshttpsonly.constraints.gatekeeper.sh             2022-09-14T17:50:52Z
k8sazurev1serviceallowedports.constraints.gatekeeper.sh          2022-09-14T18:35:41Z
k8sazurev2blockhostnamespace.constraints.gatekeeper.sh           2022-09-14T17:50:42Z
k8sazurev2containerallowedimages.constraints.gatekeeper.sh       2022-09-14T17:50:41Z
k8sazurev2noprivilege.constraints.gatekeeper.sh                  2022-09-14T17:50:41Z
k8sazurev2volumetypes.constraints.gatekeeper.sh                  2022-09-14T18:20:41Z
k8sazurev3allowedcapabilities.constraints.gatekeeper.sh          2022-09-14T17:50:45Z
k8sazurev3allowedseccomp.constraints.gatekeeper.sh               2022-09-14T18:20:45Z
k8sazurev3allowedusersgroups.constraints.gatekeeper.sh           2022-09-14T18:35:41Z
k8sazurev3containerlimits.constraints.gatekeeper.sh              2022-09-14T17:50:41Z
k8sazurev3hostfilesystem.constraints.gatekeeper.sh               2022-09-14T18:50:41Z
k8sazurev3hostnetworkingports.constraints.gatekeeper.sh          2022-09-14T17:50:50Z
k8sazurev3noprivilegeescalation.constraints.gatekeeper.sh        2022-09-14T17:50:47Z
k8sazurev3readonlyrootfilesystem.constraints.gatekeeper.sh       2022-09-14T18:20:42Z
meshtlsauthentications.policy.linkerd.io                         2023-11-10T15:39:04Z
networkauthentications.policy.linkerd.io                         2023-11-10T15:39:04Z
secretproviderclasses.secrets-store.csi.x-k8s.io                 2022-08-25T13:53:14Z
secretproviderclasspodstatuses.secrets-store.csi.x-k8s.io        2022-08-25T13:53:14Z
serverauthorizations.policy.linkerd.io                           2023-11-10T15:39:04Z
servers.policy.linkerd.io                                        2023-11-10T15:39:04Z
serviceprofiles.linkerd.io                                       2023-11-10T15:39:04Z
volumesnapshotclasses.snapshot.storage.k8s.io                    2022-08-25T13:34:17Z
volumesnapshotcontents.snapshot.storage.k8s.io                   2022-08-25T13:34:17Z
volumesnapshots.snapshot.storage.k8s.io                          2022-08-25T13:34:17Z
bhanu@Azure:~$

bhanuprakash-1 · 2023-11-28T06:26:52Z

bhanuprakash-1
Nov 28, 2023
Author

Hi @mateiidavid gentle reminder.

0 replies

bhanuprakash-1 · 2023-11-28T06:34:34Z

bhanuprakash-1
Nov 28, 2023
Author

Output of linkerd upgrade --crds:
linkerdUpgradeCrdsOutputCanary.txt

3 replies

bhanuprakash-1 Dec 18, 2023
Author

Root cause for the Primary issue is found.
There was some bug in Azure Kubernetes service's extension, specifically Azure VM Extension that we are using. This resulted in the pods not getting service account K8s tokens after their expiry, without which they can't prove their identity, hence the linkerd-proxy pods were unable to refresh their leaf certificates by calling linker control plane pods.

This resulted in linkerd-proxy containers having expired certs and no way of refreshing those certs and so went into bad state.

Not a linkerd bug.

xolott Jun 19, 2024

Hi @bhanuprakash-1 Do you have any more details here? Links to that info?

We are facing a similar issue. After one year of running without issues, our cluster went completely down because our pods reported Bad Certificate.

Until now, I was under the idea that it was a linkerd limitation with the trust anchor. Our trust anchor duration is 1-year. I restarted the control plane and every service mesh member to fix the issue, but it looks like we would have to manually restart the service mesh before the one-year period so they can load the new trust anchor.

We are also using AKS, so I would appreciate it if you could provide more information about this bug and the VM extension you were using.

bhanuprakash-1 Jun 20, 2024
Author

Hi @xolott

The extension that I was referring to is an internal Microsoft extension called "azsecpack". It was an incident and resolved then itsel. I don't think it might be the cause for your issue.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Linkerd proxy-to-proxy container calls failing with reason "Invalid Peer Certificate CertExpired" despite root and issuer cert unexpired. #11581

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 4 comments 9 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Linkerd proxy-to-proxy container calls failing with reason "Invalid Peer Certificate CertExpired" despite root and issuer cert unexpired. #11581

bhanuprakash-1 Nov 6, 2023

Replies: 4 comments · 9 replies

mateiidavid Nov 7, 2023

bhanuprakash-1 Nov 7, 2023 Author

bhanuprakash-1 Nov 7, 2023 Author

bhanuprakash-1 Nov 7, 2023 Author

mateiidavid Nov 7, 2023

bhanuprakash-1 Nov 7, 2023 Author

sharmakhushboo Nov 13, 2023

bhanuprakash-1 Nov 14, 2023 Author

bhanuprakash-1 Nov 28, 2023 Author

bhanuprakash-1 Nov 28, 2023 Author

bhanuprakash-1 Dec 18, 2023 Author

xolott Jun 19, 2024

bhanuprakash-1 Jun 20, 2024 Author

bhanuprakash-1
Nov 6, 2023

Replies: 4 comments 9 replies

mateiidavid
Nov 7, 2023

bhanuprakash-1 Nov 7, 2023
Author

bhanuprakash-1 Nov 7, 2023
Author

bhanuprakash-1
Nov 7, 2023
Author

bhanuprakash-1 Nov 7, 2023
Author

bhanuprakash-1 Nov 14, 2023
Author

bhanuprakash-1
Nov 28, 2023
Author

bhanuprakash-1
Nov 28, 2023
Author

bhanuprakash-1 Dec 18, 2023
Author

bhanuprakash-1 Jun 20, 2024
Author