Certificate Not Connected?

[jonrogoff]

I initially had a similar issue to #21. I was able to get past that by changing the insecureEdgeTerminationPolicy from Redirect to Allow.

Now I get the following:
`watching routes with selector butter.sh/letsencrypt-managed=yes

  | Processing route /oapi/v1/namespaces/XXX/routes/XXX.com with domain XXX.com.
  | unable to load certificate
  | 140546834421664:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:703:Expecting: TRUSTED CERTIFICATE
  | Getting new certificate for XXX.com
  | Adding well-known route.
  | calling dehydrated with domain name 'XXX.com'
  | # INFO: Using main config file /usr/share/letsencrypt-container/config
  | Using private key /etc/openshift-letsencrypt/account-key instead of account key
  | Processing XXX.com
  | + Signing domains...
  | + Generating private key...
  | + Generating signing request...
  | + Requesting challenge for XXX.com...
  | + Already validated!
  | + Requesting certificate...
  | + Checking certificate...
  | + Done!
  | + Creating fullchain.pem...
  | Defer deploying certificate for routes.
  | + Done!
  | Running exit_hook
`
However, when I access https://XXX.com I still receive either an insecure website warning or a mismatched certificate notification depending on which browser I am using. What am I missing?

[ibotty]

Sorry, for not answering earlier.

Can you please confirm, that there is a certificate in your route:

 oc get route XXX.com -o yaml

This should include the certificate and private key. Does it?

[jonrogoff]

No. It has neither.

I double checked and the letsencrypt serviceaccount has been bound to the edit role. However, when I check oc policy who-can edit route it only includes system:admin, system:cluster-admins, and system:masters. I'd prefer not to bind the letsencrypt serviceaccount to the admin role, unless absolutely necessary. Do I need to add do something else to allow the edit role or the letsencrypt serviceaccount to edit routes or am I comparing apples and oranges?

On a separate note, I'm curious if this project truly requires 512 MiB for each container. Could I safely scale each container back to the minimum of 255 MiB?

[ibotty]

Hi, I am sure, it does not need 512MiB in total ;). You can surely grant it only the minimum.

Regarding your route not getting updated can you please check that the letsencrypt service account can patch routes?

 oc policy who-can patch routes -n the_right_namespace

If this does not include the letsencrypt service account, can you double check, that you granted it in the right namespace?

[jonrogoff]

For now, I only have 1 namespace. I need to create more, but I want to get 1 working first.

I did confirm that system:serviceaccount:XXX:letsencrypt is in the Users list for patching routes.

[ibotty]

Is there no log entry after "Running exit_hook"? Does the pod exit ungracefully? (What does oc describe po letsencrypt-... say?

[jonrogoff]

Nothing after "Running exit_hook". Pod is running fine. The describe is very long but essentially says that watcher, cron, and nginx are all running.

[ibotty]

Is there no indication of any containers having restarted?

[jonrogoff]

Nope. I was getting that when the insecureEdgeTerminationPolicy was set to Redirect rather than Allow. At that time, the cron container kept restarting, which occasionally forced a restart of the entire pod. However, since changing the policy to Allow it all stays up and running nicely.

[ibotty]

Can you please rsh to the cron container, and delete contents below /var/lib/letsencrypt-container:

rm -r /var/lib/letsencrypt-container/*

And see, whether it changes anything?

[jonrogoff]

Now I'm getting this when requesting the certificate:

`
Details:

  | {
  | "type": "urn:acme:error:rateLimited",
  | "detail": "Error creating new cert :: too many certificates already issued for exact set of domains: XXX.com",
  | "status": 429
  | }
`

I suppose I'll have to check again later.

[ibotty]

Unfortunately yes. Please do comment, Wien it works again. Alternatively you might switch to the test acme server.

[jonrogoff]

I created a new subdomain for testing. Now I get this:

Processing XXX.com
--
  | + Signing domains...
  | + Generating private key...
  | + Generating signing request...
  | + Requesting challenge for XXX.com...
  | Running deploy_challenge hook
  | + Responding to challenge for XXX.com...
  | Deleting well-known route.
  | + Challenge is valid!
  | + Requesting certificate...
  | + Checking certificate...
  | + Done!
  | + Creating fullchain.pem...
  | Defer deploying certificate for routes.
  | + Done!
  | Running exit_hook

As before, attempting to access XXX.com complains because the certificate has the wrong name.

[ibotty]

That's an error I did not encounter before. Can you try to patch the route manually? Please don't delete the pod (so the certificate is still in the filesystem). I will write instructions Monday, if you don't know how to generate the patch and apply it manually (See common.sh for the source).

[jonrogoff]

I'm familiar with starting a new build, but that creates a new deployment/pod. So I may need to be pointed in the right direction for how to patch the pod directly. I know linux so if the patch is applied via ssh, it should be easy enough.

[ibotty]

Sorry, that was a little terse. I will write tomorrow if time permits, but in short, you don't have to change the deployment (or stop the running pod) at all. To add the certificate to the route, openshift-letsencrypt patches the route (see common.sh). I wanted to do that manually, so we can see explicitly what's wrong.

[malagant]

Same here. Would be glad to help out.
Patching the route still doesn't seem to work after getting the certs.
Any progress on this?

[jonrogoff]

It has been a busy week. I'm just getting back to this today. After reviewing common.sh, I ran common.sh then manually called add_certificate_to_route <mydomain> /oapi/v1/namespaces/<myproject>/routes/<myroute>. It returned without an error. However, the info specified in the data section was not added to the yaml for myroute.

I edited common.sh to echo the data parameter passed to patch_route. I then attempted to manually edit my route to add that data and received errors that spec.tls.certificate and spec.tls.key are immutable.

I am using OpenShift Online Pro (not Starter) so I should be able to specify certificates. I therefore tried deleting and recreating my route using the key and certificate info from the data parameter and got this error:
Reason: - spec.tls.certificate: Invalid value: 'redacted certificate data': Could not read any certificates - spec.tls.key: Invalid value: 'redacted key data': tls: failed to find any PEM data in certificate input.

I notice that the data actually includes 2 certificates within the value for spec.tls.certificate. They have the line http://cert.int-x3.letsencrypt.org/ in between them. Is this correct?

[malagant]

#26 is IMO also related to this, which makes me believe that origin 3.6 introduces an API change. I will dig deeper into this and keep you informed about any progress I make.
@jonrogoff thanks for the information.

[ibotty]

Note, that I won't be able to take care of any support requests for at least a month starting tomorrow. I am sorry.

[hypery2k]

can confirm that's related to #openshift/origin#18397. It's a right issue