Update cryptography version?

[ohemorange]

As @bmw commented in #10088 (comment):

our normal approach to choosing minimum dependency versions is described at https://eff-certbot.readthedocs.io/en/stable/contributing.html#choosing-dependency-versions

in this particular case, it looks like RHEL 9 + EPEL has cryptography 36.0.1 and pyopenssl 21.0.0. how hard would it be to make things work using nothing later than these versions? us requiring a newer version would essentially mean cutting off that channel from any future updates to certbot which would ideally be avoided

But, as I said at #10097 (comment):

Between #10097, #10088, and certbot/josepy#196, and #10103, I'm seeing several places where having a newer cryptography package would be nice. I understand that we want to minimize work for packagers, but I feel like "package a newer cryptography while you're packaging a newer certbot" isn't too excessive of an ask.

And snap is available on RHEL so even if this became too big of a burden that packagers were unwilling to do it, RHEL users would not be precluded from having newer certbot versions.

This issue is for discussing the potential change and potentially tracking the work. We'll need at least 42.0.0 for not_valid_after_utc and not_valid_before_utc.

[wgreenberg]

i'm generally in favor of this, though worth noting (as you mentioned in the josepy issue) that updating cryptography also means updating pyOpenSSL, which may add to the complexity of the upgrade.

[ohemorange]

I take your point, but how about we instead then just drop pyopenssl? Seems like we're on our way as it is.

[bmw]

i'm personally generally supportive of us declaring whatever minimum dependency versions makes things easiest for us and our own distribution methods (snap, docker, pip, etc.) and only consider doing anything else if significant demand arises

to share my understanding of the downsides to make sure we're all on the same page though, i'm pretty confident EPEL packagers will be unwilling or unable upgrade cryptography because its packaged directly in RHEL which has very strict rules for backwards compatibility compared to EPEL. i think we may come to find this unfortunate when for example let's encrypt starts issuing shorter lived certs by default and the certbot package there isn't updated to handle that

with that said, we have the same problem shipping updates to stable versions of debian and ubuntu which are more popular and fedora seems to be holding back on updating users to certbot 3.0, likely due to our major version number changing. they did update 1.0 users to 2.0 in the past so we could potentially convince them to push users to 3.0, but at the end of the day it's their decision, not ours

if we do decide to go forward with this change, i think we may want to consider updating https://eff-certbot.readthedocs.io/en/stable/contributing.html#choosing-dependency-versions since we'll no longer be maintaining compatibility with EPEL9. the existence of that text is largely why i initially pushed back on upgrading cryptography in the first place

[ohemorange]

CentOS 9 Stream BaseOS only has 36.0.1, but CentOS AppStream seems to be including newer versions, currently up to 41.0.7. It seems the whole point of AppStream is to include newer versions of packages. Is the problem that they don't have 43 just yet, or that we can't rely on AppStream packages?

[bmw]

the problem as i understand it is that certbot in EPEL9 cannot rely on appstream packages

centos used to be downstream from RHEL, essentially taking out all of RHEL's commercial branding and repackaging it for free, but red hat changed things a bit ago by introducing things like appstream and making centos a testing ground for future versions of RHEL. this caused a mass exodus from centos with people who want a free RHEL now using rocky linux or almalinux. my understanding is users of these distros cannot or do not want to use appstream

[ohemorange]

The contents of that link say that EPEL is built against appstream though? How can it be built against appstream and not have those other users have appstream?

[bmw]

my understanding is "EPEL 9 Next" is built against the centos appstream you're seeing, but "EPEL 9", where we're currently packaged, is not. i believe "rhel-9-for-*-appstream-rpms" is a different thing than the centos appstream packages you're seeing. if we wanted to play around with this, i think both almalinux and rocky linux images are now available on digital ocean

with that said, here's some stats on our current users by OS: https://gist.github.com/bmw/f71accbad8ac386793c44fc6260e428f

adding up the RHEL 9 based distros i see, i'm counting 73,500 w/o centos 9 stream and 89,636 with it. i think centos 9 is worth counting because while i think it may be possible to get certbot packaged in "EPEL 9 next" to provide an upgrade path to centos 9 users, i believe that hasn't happened yet and that users would have to also configure their system to track the EPEL 9 next repo to get an upgrade

this is a slight undercount because there's a long tail of fringe OSes many of which are based on top of another system, but that means we're talking about roughly 1.6% - 2% of our users here. that's certainly not nothing, but unless us keeping old cryptography support is trivial, i personally think it's small enough that it's fine to make our own lives easier, do the upgrade, and update our docs/policy here

[ohemorange]

I don't think they're different; pretty sure "EPEL 9 Next" is the staging version of EPEL and "centos appstream" is the staging version of rhel-9-for-*-appstream-rpms.

from https://docs.fedoraproject.org/en-US/epel/epel-about-next/:

"Next" correctly describes the purpose of the repository, which is providing packages compatible with the next minor release of RHEL.

[ohemorange]

Ok, I just went ahead and made an almalinux droplet. Appstream is enabled by default, I literally just logged in and we have:

[root@erica-almalinux-s-1vcpu-512mb-10gb-sfo3-0 ~]# dnf search --showduplicates cryptography
Last metadata expiration check: 0:04:37 ago on Tue Jan  7 19:24:31 2025.
============================ Name & Summary Matched: cryptography ============================
python3-cryptography-36.0.1-4.el9.x86_64 : PyCA's cryptography library
python3.11-cryptography-37.0.2-6.el9.x86_64 : PyCA's cryptography library
python3.12-cryptography-41.0.7-1.el9.x86_64 : PyCA's cryptography library

[ohemorange]

...and now I see that I could have just checked this in pkgs.org, oh well.

[bmw]

ah you're right about the relationship between centos's appstream and the rhel appstream repos. sorry for the misinformation and thanks for correcting my understanding there

that makes the situation slightly better, but i still personally feel pretty confident certbot in EPEL 9 will not see any more updates if we make this change (although it may not anyway due to aforementioned hesitation about our major version bump). the problem i see now digging into it a bit more is that these other versions of cryptography are packaged for a different version of python and EPEL maintainers would need to repackage all certbot dependencies for this other python version in order for this to work. poking around a little bit, it seems that all/most of our dependencies that are in EPEL and not base RHEL or appstream are not packaged for any other python version. we were told that doing this kind of work wasn't feasible for EPEL 7 in the past at https://bugzilla.redhat.com/show_bug.cgi?id=1813670#c4

again though, i'm personally also in favor of this change. i think ~2% of our users is relatively small, EPEL may not want to update to 3.0+ anyway, we'll already need to deal with problems caused much more widely with old certbots in debian & ubuntu, and our current docs on this issue come from #9681 where in the very limited linked mattermost conversation about what we should do i also (half-heartedly) suggested we stop caring about EPEL packages

[ohemorange]

That response was before they switched to the stream/appstream model, so there's a chance it could be different now. I do think that regardless of anything else it's important to understand what can and cannot be packaged. It may still be too much work to be feasible, but "there is no easy way currently to build certbot against these [additional python 3] modules" may no longer be true.

Then secondly, there's the question of "is it irrelevant anyway because 3.0 is never going into EPEL 9?" That would certainly make the decision easier, but I personally can't tell if that's a "just haven't gotten around to it" or a "not going to;" 2.10 was skipped entirely, and 2.7 and 2.8 went into Fedora numbered versions but not EPEL.

As to the larger question of not caring about EPEL, I agree it would be nice for us! I don't think 2% of our users is that small, but I also think that given the appstream ecosystem, if EPEL packagers really want to they do now have a way to package an updated version of certbot. So I'm also for no longer officially supporting EPEL, especially since our official instructions do say to use snap, it's not likely users on those system will be left behind entirely if they need to upgrade.

[bmw]

that's fair!

it seems like we're high level agreement: we should go ahead and update our minimum cryptography version and not expend energy right now trying to maintain compatibility for older packages for EPEL