Certificate signed by unknown authority while installing with helm

[neurosis71]

Hi guys,

I'm encountering some trouble installing linkerd on my EKS cluster because of my certificates.

Problem

The main error I see is this one :

x509: certificate signed by unknown authority (possibly because of \"x509: ECDSA verification failure\" while trying to verify candidate authority certificate \"root.linkerd.cluster.local\")

I've been scratching my head around this problem for a couple of days now and I'm running out of clues.
Is there a way to resolve or circumvent this problem somehow ?

Technical details

I'm installing linkerd using this command

helm install linkerd2 \
  -f config.yaml \
  -n linkerd \
  --set-file identityTrustAnchorsPEM=certificat/ca.crt \
  --set-file identity.issuer.tls.crtPEM=certificat/issuer.crt \
  --set-file identity.issuer.tls.keyPEM=certificat/issuer.key \
  --set identity.issuer.crtExpiry=$(date -d '+8760 hour' +"%Y-%m-%dT%H:%M:%SZ") \
  --set policyValidator.externalSecret=true \
  --set-file policyValidator.caBundle=certificat/issuer.crt \
  --set proxyInjector.externalSecret=true \
  --set-file proxyInjector.caBundle=certificat/issuer.crt \
  --set profileValidator.externalSecret=true \
  --set-file profileValidator.caBundle=certificat/issuer.crt \
  linkerd/linkerd2

And here is the config file I use

# This values.yaml file contains the values needed to enable HA mode.
# Usage:
#   helm install -f values.yaml -f values-ha.yaml

enablePodAntiAffinity: true

# proxy configuration
proxy:
  resources:
    cpu:
      request: 100m
    memory:
      limit: 250Mi
      request: 20Mi

# controller configuration
controllerReplicas: 3
controllerResources: &controller_resources
  cpu: &controller_resources_cpu
    limit: ""
    request: 100m
  memory:
    limit: 250Mi
    request: 50Mi
destinationResources: *controller_resources

# identity configuration
identityResources:
  cpu: *controller_resources_cpu
  memory:
    limit: 250Mi
    request: 10Mi

# heartbeat configuration
heartbeatResources: *controller_resources

# proxy injector configuration
proxyInjectorResources: *controller_resources
webhookFailurePolicy: Fail

# service profile validator configuration
spValidatorResources: *controller_resources


#Required to set automatic rotation of control plane TLS credentials
#https://linkerd.io/2.11/tasks/automatically-rotating-control-plane-tls-credentials/
installNamespace: false
namespace: linkerd


identity:
  # -- If the linkerd-identity-trust-roots ConfigMap has already been created
  externalCA: false
  issuer:
    scheme: kubernetes.io/tls

    # -- Amount of time to allow for clock skew within a Linkerd cluster
    clockSkewAllowance: 20s

    # -- Amount of time for which the Identity issuer should certify identity
    issuanceLifetime: 24h0m0s

Shortly after the installation is done the created pods go in CrashLoopBackOff

linkerd        linkerd-destination-7bfdb4dc7-xbl5w                      0/4     CrashLoopBackOff   8 (2m6s ago)    10m
linkerd        linkerd-destination-7bfdb4dc7-zfdlc                      0/4     CrashLoopBackOff   8 (2m9s ago)    10m
linkerd        linkerd-destination-7bfdb4dc7-zqqw8                      0/4     CrashLoopBackOff   8 (2m8s ago)    10m
linkerd        linkerd-identity-6885b49f5c-cjfvv                        0/2     CrashLoopBackOff   6 (4m8s ago)    10m
linkerd        linkerd-identity-6885b49f5c-njsbf                        0/2     CrashLoopBackOff   6 (4m10s ago)   10m
linkerd        linkerd-identity-6885b49f5c-zgm8l                        0/2     CrashLoopBackOff   6 (4m2s ago)    10m
linkerd        linkerd-proxy-injector-745654f574-4b8r7                  0/2     CrashLoopBackOff   4 (2m6s ago)    10m
linkerd        linkerd-proxy-injector-745654f574-5pd9k                  0/2     CrashLoopBackOff   4 (2m6s ago)    10m
linkerd        linkerd-proxy-injector-745654f574-bh7cs                  0/2     CrashLoopBackOff   4 (2m6s ago)    10m

When I get the log from the identity pod

# kubectl logs -n linkerd linkerd-identity-6885b49f5c-cjfvv identity 

time="2022-05-27T13:29:27Z" level=info msg="running version stable-2.11.2"
time="2022-05-27T13:29:27Z" level=fatal msg="Failed to initialize identity service: failed to verify issuer credentials for 'identity.linkerd.cluster.local' with trust anchors: x509: certificate signed by unknown authority (possibly because of \"x509: ECDSA verification failure\" while trying to verify candidate authority certificate \"root.linkerd.cluster.local\")"

linkerd check returns this

#linkerd check

Linkerd core checks
===================

kubernetes-api
--------------
√ can initialize the client
√ can query the Kubernetes API

kubernetes-version
------------------
√ is running the minimum Kubernetes API version
√ is running the minimum kubectl version

linkerd-existence
-----------------
√ 'linkerd-config' config map exists
√ heartbeat ServiceAccount exist
√ control plane replica sets are ready
√ no unschedulable pods
× control plane pods are ready
    pod/linkerd-destination-7bfdb4dc7-zfdlc container sp-validator is not ready
    see https://linkerd.io/2.11/checks/#l5d-api-control-ready for hints

Status check results are ×

I followed your documentation here and used the exact same commands.

Thank you for your help guys !

[olix0r]

The main error I see is this one :

x509: certificate signed by unknown authority (possibly because of \"x509: ECDSA verification failure\" while trying to verify candidate authority certificate \"root.linkerd.cluster.local\")

Where do you see this error?

I'm not sure if this is related but

  --set policyValidator.externalSecret=true \
  --set-file policyValidator.caBundle=certificat/issuer.crt \
  --set proxyInjector.externalSecret=true \
  --set-file proxyInjector.caBundle=certificat/issuer.crt \
  --set profileValidator.externalSecret=true \
  --set-file profileValidator.caBundle=certificat/issuer.crt \

You almost definitely do not want to use the issuer certificate as the webhook CA bundle. These are unrelated.

I see you are using externalSecret=true -- are you planning to use cert manager? Or what is responsible for writing these secrets?

[neurosis71]

Hi @olix0r, Thank you for your response !

The error is seen in the identity pod using this command :

kubectl logs -n linkerd linkerd-identity-6885b49f5c-cjfvv identity 

Yes I'm trying to setup cert-manager to rotate the certificates automatically so I followed those two pages to setup cert-manager and all :

https://linkerd.io/2.11/tasks/automatically-rotating-control-plane-tls-credentials/

https://linkerd.io/2.11/tasks/automatically-rotating-webhook-tls-credentials/

So I used this command to generate my control plane certificate :

step certificate create root.linkerd.cluster.local ca.crt ca.key \
  --profile root-ca --no-password --insecure

And this one for the webhook cert :

step certificate create webhook.linkerd.cluster.local issuer.crt issuer.key \
  --profile root-ca --no-password --insecure --san webhook.linkerd.cluster.local

For some reason I managed to install linkerd via the cli using this command and everything works fine with this method.

linkerd install \
  --identity-trust-anchors-file certificat/ca.crt \
  --identity-issuer-certificate-file certificat/issuer.crt \
  --identity-issuer-key-file certificat/issuer.key \
  --set policyValidator.externalSecret=true \
  --set-file policyValidator.caBundle=certificat/issuer.crt \
  --set proxyInjector.externalSecret=true \
  --set-file proxyInjector.caBundle=certificat/issuer.crt \
  --set profileValidator.externalSecret=true \
  --set-file profileValidator.caBundle=certificat/issuer.crt \
  --ha \
  | kubectl apply -f -

I don't quite get the difference with the two methods as they have the same setup.

Hope this helps understand better.

Thank you for your help !

[adleong]

Hi @neurosis71! It looks like you're using the same file (issuer.crt) as both the identity issuer and the webhook ca bundle which probably won't work. There are a lot of moving pieces here so I'd recommend following the instructions and commands in https://linkerd.io/2.11/tasks/automatically-rotating-webhook-tls-credentials/ as closely as possible.

[daro1337]

I have exactly the same error as @neurosis71

@neurosis71 I've inspect linkerd install output, and identity:issuer:tls:crtPEM == identityTrustAnchorsPEM
now, I'm up and running

[rufreakde]

@daro1337 you mean you had to manually set in the helm chart identity:issuer:tls:crtPEM == identityTrustAnchorsPEM for both to be the same certificate value?

In the guide I see that the values should be different no?

helm install linkerd-control-plane -n linkerd
--set-file identityTrustAnchorsPEM=ca.crt
--set-file identity.issuer.tls.crtPEM=issuer.crt
--set-file identity.issuer.tls.keyPEM=issuer.key
linkerd/linkerd-control-plane

[KingLeak95]

I also have the exact same error, when it was working a few days ago