Expired certificate in CA bundle makes api/health/checks/certificate-expiration return a failure even if the CA is not used by server certificate

[kmorwath]

Describe the bug

I have a RabbitMQ 4.0.4 server running on Ubuntu 22 LTS. It is set to use the CA Bundle in /etc/ssl/certs/ca-certificartes.crt for TLS. One of the certificates - Security_Communication_Root_CA.pem - expired (notAfter=Sep 30 04:20:49 2023 GMT). When invoking api/health/checks/certificate-expiration, the result was:

{
    "status": "failed",
    "reason": "Certificates expiring",
    "expired": [
        {
            "node": "rabbit@myservice",
            "port": 5671,
            "protocol": "amqp/ssl",
            "interface": "::",
            "cacertfile": "/etc/ssl/certs/ca-certificates.crt",
            "certfile": "/etc/ssl/certs/myservice.domain.internal.pem",
            "cacertfile_expires_on": [
                "2023-09-30 4:20:49"
            ],
            "certfile_expires_on": [
            ]
        }
    ]
}

But myservice.domain.internal.pem was generated by a different CA which is still valid. It looks any certificate in the CA bundle is tested, not only the CA which generated the actual certificate used by the RabbitMQ server. That makes monitoring application report a certificate is expired even when it is not.

After Removing the expired CA the correct result was returned.

Reproduction steps

1. Configure RabbitMQ server to use a valid certificate issued by a valid ca in /etc/ssl/certs/ca-certificates.crt
2. Ensure an expired certificare is in /etc/ssl/certs/ca-certificates.crt
3. call /api/health/checks/certificate-expiration
4. status "failed" is returned.

Expected behavior

status: "ok"

Additional context

Ubuntu 22 LTS fully updated to date.

[michaelklishin]

@kmorwath you haven't provided any evidence of a bug.

Our community support policy very clearly states that we will not troubleshoot TLS for non-paying users. We have no details on how to reproduce this behavior in any case.

[lukebakken]

I have re-opened this discussion and have deleted comments that are not helpful.

@kmorwath - at the very least you could provide a CA bundle that exhibits this behavior. Yes, I could spend time doing that but it's time I could be using to investigate this behavior.

Currently, the expiration check looks at all certs in the CA bundle and if any are expired, returns failure. To improve on that, we would have to determine which cert or certs are actually being used by RabpitMQ, which may not be worth the effort.

Nearly every RabbitMQ user uses a CA "bundle" consisting of one root cert, usually that of their own cert infrastructure.

Thanks.

[michaelklishin]

And our team's preferred way to producing certificate chains for troubleshooting is with tls-gen, with modifications to profiles and their openssl.cnf not only acceptable but often expected to re-create more compelex real world cases.

Specifically overriding the certificate expiration date is a matter of tweaking one environment variable or one openssl.cnf key.

[kmorwath]

Here a Ubuntu 22 LTS default CA bundle containing an expired CA certificate (it's Security Communication RootCA1). ca-certificates.zip.

If needed, add the root CA of the RabbitMQ certificate at the end of the list. Then point "cacertfile" to the bundle, and call "api/health/checks/certificate-expiration".

Nearly every RabbitMQ user uses a CA "bundle" consisting of one root cert, usually that of their own cert infrastructure.

I understand it - and why it could have not been found before. But our deployment process requires CAs to be added to the OS default store locations, and application then configured to use that.

Currently, the expiration check looks at all certs in the CA bundle and if any are expired, returns failure.

Well, this is a bug. There's no reason to process CAs outside the RabbitMQ certificate chain. A CA bundle may contain expired certificates. Each certificate contains its issuer data - allowing to identify and check it until the self-signed root CA is reached.

Otherwise the specific, non-standard RabbiMQ behaviour should be documented, and that "cacertfile" needs to point to a single certificate too.

Update

Probably this is the root cause of the issue: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1063093 (I was using Ubuntu, but I guess it gets CAs upstream from Debian or Mozilla) Although, as message #10 says "It doesn't actually matter, though, and it'll be gone next time we pull from mozilla.". Still RabbitMQ should not return a false failure - a CA may expire any time, it should impact only certificates down its chain - RabbitMQ was the only TLS application we're using to have issues.

[lukebakken]

"cacertfile" needs to point to a single certificate too

No, it doesn't need to, it's just that is how almost everyone uses RabbitMQ.

More than likely, changing the certificate expiration check to only verify in-use certs will not be worth the effort.

[michaelklishin]

@kmorwath this is not a bug. There is no law of physics that says which certificates in a chain, or CA certificates for that matter, must or must not be inspected by an optional health check.

The whole point of this health check — which is entirely optional — is to WARN you. Having expired certificates is not a typical condition in most deployments.

This health check, even though unintentionally, does you a services in warning you. If you don't like that, don't build your own health check in a plugin or as a tool external to RabbitMQ.

Even if it was a bug, you would be most welcome to contribute a fix and find out in the process that it is not at all obvious what CA certificate in a file is relevant and which is not.

You get the software for free, and you are the only person who has this problem. So either help fix it by providing a very detailed algorithm and test cases.

[michaelklishin]

@kmorwath I am looking forward to a detailed algorithm with tests, ideally with an implementation, in a PR or a discussion under Ideas. Or simply stop using that optional health check if it does not work for you and move on.

We do not consider this catastrophic bug that affects one person to be worth our small team's time.

[michaelklishin]

Investigation

After looking at a number of popular data services and HTTP servers, I conclude that they often
do not provide a similar health check, or its behavior is limited to process startup and a logged warning, and usually the documentation is limited to a mention of the warning.

There may be exceptions but the norm is to tell the user to figure certificate validation out
however they like, including a verification of expiration.

In addition, a very typical feature in the industry is to ask the user to provide a bundle full
of relevant certificates and use them plus a number of trusted roots that come from an OS-specific source. There is no other commonly used mechanism for determining which certificates are "relevant"
and which are not.

Therefore, @kmorwath's environment is not a typical case, and the claim that
determining what certificates are "relevant" from an arbitrary list of certificates
found in a PEM is at least not a solved problem in the industry.

Section 6 of RFC 5280 does
not really cover arbitrary certificates present in the chain in any reasonable detail.
The algorithm is based on traversing a chain of certificates where it is a known fact
that certificate N is issued by a certificate N-1 (the previous one) in the chain.

Verifying such relation between every pair of certificates to find out expired certificates
should be possible but most tooling does not do that, or does not provide a similar
health check to begin with, for example:

• In pg-cert-check
• According to mongosh documentation and source code
• For etcd
• For nginx

and so on.

So what RabbitMQ's health check does seems to be a very standard thing to do
across a range of different a popular tools.

Documentation Updates

As of #13037 (which will ship in 4.1.0 and 4.0.6,
the behavior of this health check is documented in

1. The HTTP API reference page, where we have more space for details
2. In rabbitmq-diagnostics help check_certificate_expiration

In [1] we now explicitly recommend not using this health check in environments where either

A. An expired certificate in one of the bundles is a normal operating condition
B. Some certificates found in those files are not related to RabbitMQ operations in any way

An Alternative Solution

An alternative health check can be implemented as a very small command line tool
that would do something like this:

a. Load all relevant certificate bundles (how to determine which files are relevant outside of RabbitMQ is left to the reader to figure out)
b. Parse them
c. For each certificate, determine if it is "RabbitMQ-relevant" (how to do that is left to the reader to figure out using section 6 in RFC 5280 as a rule of thumb)
d. For every "RabbitMQ-relevant" certificate, verify the expiration date

This tool can be built using any programming language that has a PKI/x509 library, which is anything
even distantly popular.

I consider this case to be closed. If someone wants to provide an example of an alternative
algorithm with tests, knock yourself out.