GCC versions side-by-by-side support on Ubuntu images

[AlenaSviridenko]

Hi there,

As you may know, we support several GCC versions installed side-by-side on Ubuntu images. It went well until we installed GCC-11 with experimental toolchain in the scope of #3280.

After that we received a lot of reports that it breaks current customers due to incompatibility of libstd++ in GCC-11 and the earlier versions (#3432, #3454, #3376).

Our first action to mitigate the impact — remove GCC-11 from Ubuntu images. But at the same time we understand that this is a highly-requested tool, and new versions should be pre-installed on the images.

So, we'd like to start a discussion with experts in C++ to understand what is the best strategy in supporting several side-by-side GCC versions, is it even possible, how to isolate them to avoid issues like this one.

We will appreciate your opinion on it. Thanks!

[AlenaSviridenko]

Hi @EwoutH, @jesec,
would be really great if you could share your thoughts on it.

Thanks!

[jesec]

See my previous comments: #3376 (comment), #3451 (comment).

The current version of GCC 11 is utterly useless if it can't provide binary compatibility with the base operating system. Even if you somehow decides to use this GCC 11, do you really want to ask your customers: OK we use GCC 11 to compile binaries, so to run our binaries you have to install the experimental version of standard C++ library from an unsupported PPA, which may f your entire environment up in the process?

The toolchain is not yet production ready, and we should not allow this experimental piece of software to render the entire environment unusable.

A production ready toolchain provides binary compatibility with the base operating system. Ubuntu maintainers will take special care of that. The current issue is likely just temporary. There is no point to work on any "harmony" or "isolation", as a non-production-ready toolchain simply cannot be used at all.

You can check out the manual of RHEL's devtoolset to learn more about how toolchain maintainers made newer compilers available to older OS while maintaining binary compatibility. I believe Ubuntu maintainers are doing the same, and they are still working on the GCC 11.

My opinion: Simply wait for the Ubuntu maintainers to deliver a proper production-ready toolchain that is intended to work with Ubuntu 20.04. Meanwhile, Get rid of the experimental toolchain that is tainting the environment. Remove unsupported PPA, and only pull toolchains from the archive. If a newer version of GCC is not available in the archive, it can not be used at all.

Developers that rely on newer features have to install newer toolchains at their own risk (namely, may not assume that the binary is compatible with the base OS). It is also an option to static link in libstdc++ and libgcc, but that creates other issues (bigger size, security, exception catching, etc.). or ask your customers to install unsupported libstdc++ on their system.

Alternatively, with engineering efforts, one may replicate EL's devtoolset (new compiler that retains compat with base OS) on Ubuntu platforms.

[jesec]

Edit:

Unfortunately, I was wrong. After #3432 (comment), I took a deeper look at the situation. Previously I made an assumption that GCC 9 and 10 are "production-ready" on Ubuntu 18.04 AND Ubuntu 20.04. This is incorrect. On Ubuntu 18.04, which ships libstdc++ 8, they taint the environment in a similar fashion (by depending on non-supported libstdc++ 9+). They only work properly with Ubuntu 20.04, as Ubuntu 20.04 ships libstdc++ 10.

Due to my prior experience with RHEL devtoolset, I have also assumed that Ubuntu maintainers would backport toolchain while maintaining binary compatibility with base OS, and that they are just "still working" on GCC 11. That's wrong as well. Ubuntu maintainers do not intend to do that.

The conclusion: Only the toolchain in the Ubuntu archive maintains binary compatibility with the base OS, and can be deemed production-ready. Toolchains from the unsupported PPA do not, when the GCC version exceeds libstdc++ version of base OS, and will require end-users of built binaries to install higher versions of libstdc++. As I explained before, asking users to install a new version of libstdc++ is not a feasible option, so I updated my opinion accordingly. Alternatively, one may static link in libstdc++ and libgcc, but that creates other issues (bigger size, security, exception catching, etc.).

[michaelbprice]

I ran into this issue today. The documentation for the Ubuntu 22.04 images says that GCC 11.2.0 is installed, but experimentation and the install scripts say otherwise.

I think the best answer for these types of issues is containerization. It's not uncommon for a piece of software to install in a way that breaks other software. It is unfortunate, but common, and it is a situation that calls for the isolation that containers provide.