OpenSSL 3.2.0 update breaks connections to Aurora Postgres

[jasonmccallister]

Description:

When the container images were updated to include OpenSSL 3.2.0 in #143, subsequent deployments broke Aurora RDS connections to Postgres with the following error:

SQLSTATE[08006] [7] connection to server at "cluster.rds.amazonaws.com" (x.x.x.x), port 5432 failed: SSL error: ssl/tls alert handshake failure connection to server at "cluster.rds.amazonaws.com" (x.x.x.x.), port 5432 failed: FATAL:  no PostgreSQL user name specified in startup packet 

How to reproduce:

1. Update the container image to include OpenSSL 3.2.0, RDS connection is broken
2. Lock the specific container image sha to a previous version, connection is restored

[GrahamCampbell]

What version of postgres? Is this postgres aurora? Are you explicitly providing a CA (I know the error indicates that this has nothing to do with this... but I've seen error messages be super misleading with SSL)?

[GrahamCampbell]

Are you using week keys? The default security level has changed in openssl 3.2 from 1 to 1: https://www.openssl.org/docs/man3.2/man3/SSL_CTX_set_security_level.html.

[GrahamCampbell]

Are you able to re-build our image with -DOPENSSL_TLS_SECURITY_LEVEL=1 set when building openssl, to see if it fixes the issue?

[GrahamCampbell]

I guess I should also ask - how sure are you at that SSL is actually working before and that the client is not silently falling back to plain text? What happens if you set rds.force_ssl to 1 in the cluster parameter group?

[jasonmccallister]

What version of postgres? Is this postgres aurora? Are you explicitly providing a CA

Aurora Postgres 14.6 is the version we are using, we did not provide a CA, using the provided CA from RDS

but I've seen error messages be super misleading with SSL

Completely agree.

Are you using week keys? The default security level has changed in openssl 3.2 from 1 to 1:

We are not using TLS to connect to the RDS database, I suspect that change in OpenSSL to be to root of the cause as we were connected before the update - unable to connect after.

Are you able to re-build our image with -DOPENSSL_TLS_SECURITY_LEVEL=1 set when building openssl, to see if it fixes the issue?

I can certainly try to build the image, that would ultimately verify the issue - I'll try to look into that this week - but no promises :)

I guess I should also ask - how sure are you at that SSL is actually working before and that the client is not silently falling back to plain text?

Our connection string has not changed, only the Bref image. I can look at the param groups in the morning, but nothing changed there either.

[GrahamCampbell]

And you're sure #138 was not the cause?

[mnapoli]

@jasonmccallister is the latest version of the layers working fine now? (if it is, that would pinpoint the problem to the OpenSSL upgrade, if not that might be something else)

[jasonmccallister]

@mnapoli we set the deployments to the tagged version 2.2.10 and it has been resolved.

And you're sure #138 was not the cause?

@GrahamCampbell not 100 percent certain, but we did have deployments between those releases without any connectivity issues to RDS.

[GrahamCampbell]

This seems to be an issue that has been fixed but not yet released (and won't be released until February 2024): postgres/postgres@5dd30bb. We could apply the patch in bref when we compile libpq, allowing us to upgrade to OpenSSL 3.2, or alternatively, we'd have to wait until postgres 15.6 is released in February.

[mnapoli]

Awesome, thanks for investigating! Let's wait until the patch is released, unless we are missing something big that requires to patch ourselves.