-
Notifications
You must be signed in to change notification settings - Fork 12.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Please consider installing std.cppm
in lib++
#73089
Comments
@llvm/issue-subscribers-clang-modules Author: Boris Kolpackov (boris-kolpackov)
I've been trying out the `std` module in `libc++` with `build2` (per the [Modules in libc++](https://github.com/llvm/llvm-project/blob/main/libcxx/docs/Modules.rst) documentation) and everything seems to work without any issues, at least on simple examples. Very impressive, I must say.
There is just one snag: > Once libc++'s implementation is more mature we will reach out to build system vendors, with the goal that building the BMI files is done by the build system. In I can see two immediate questions about installing Regarding where to install, I would suggest just installing it next to the headers (e.g., into
The only objection I've heard to installing modules next to headers is that "it's not a header" and "it may be included by mistake". While both are true, we are already installing non-header files (e.g., inline/template implementation files) and nobody seems to be including them by mistake. We could also take an extra measure to preclude this by, for example, installing the modules into a subdirectory (e.g., Regarding making Thoughts? P.S. For completeness, let me also mention what I've done as a (hopefully) temporary measure in @mordante @iains @ChuanqiXu9 @dwblaikie @mathstuf |
FYI, about the position of installing modules, there is a discussion in https://lists.isocpp.org/sg15/2023/07/1971.php |
BTW, to add a data point for the "perhaps the |
Note that we don't just need the files, but also the flags to make them work as expected (namely to suppress warnings about reserved module names via |
If we start shipping CC @Bigcheese |
Great to hear it works with build2! Shipping the module files is getting at the top of my todo list and I wanted to investigate which tool vendors to contact. However last week I saw @ruoso created cplusplus/modules-ecosystem-tr#29. So I prefer to participate in that effort instead of starting a similar effort. Build2 is also on that list. Regarding the large amount of |
FWIW, this is the only extra option I had to pass to compile
How can I put this politely? SG15 still hasn't come to an official agreement on the file extension for module but it seems to have agreed on the term used for compiled modules: BMI. Though there is no agreement on what it stands for: some think it's a "binary module interface", some think it's a "built module interface", there is probably another interpretation I am not aware of. So if you wait on SG15, you may end up in a situation where build systems just run out of patience and start bundling your module interfaces. Also, FWIW, MSVC folks have decided on the location and are guaranteeing it: build2/build2#333 (comment)
Just to clarify, I didn't suggest that you get rid of them entirely, only in the installed version of |
There are at least a half dozen |
From cplusplus/modules-ecosystem-tr#29:
So it looks like that issue is not a good place to talk about the installation place for std modules. If we want to push things forward, maybe we can send another mail to SG15 with a stronger desire to get a conclusion (instead of purely discussion), or if we want, we can make the decision ourselves. (I don't have a strong opinion here) |
The specific search paths used to find module source files in general hasn't been something that SG15 has discussed so far. This is largely because that very quickly hits packaging, project layout, and distribution, which are things we care about, but are large issues that would delay getting the minimum needed for modules. I do think that discussing the limited case of where libc++ and libstdc++ should put their |
I don't see an issue with making a decision ourselves. MSVC has already decided and if we pick something sensible I don't see a reason why I think in the end it doesn't really matter where the modules are installed provided that:
The reason I personally like |
I prefer (underneath) |
I encourage you (@mathstuf and @boris-kolpackov) to take part in the SG15 discussion. Let's cut a deal here. We all want to make progress on shipping modules, and I think we would all like somewhat consistent paths across implementations to keep things simple (at least I and @Bigcheese do). Let's schedule a SG15 discussion, have it, and if there is no conclusion at the end of this SG15 discussion, then we'll do whatever we want while consulting with @jwakely to align with libstdc++. Let's at least try getting agreement within SG15 and if this doesn't seem to work, then we'll just unblock our work because I agree we need to get the ball rolling on this if we want users to start using it. |
Do any plausible examples come to mind?
There is already a convention for installing arch-specific headers into |
That's specific to Debian-based distros that follow Debian's MultiArch convention. That's not universal. |
Anything that does SIMD will have per-arch modules unless the backend is 100% isolated (which seems very unlikely to me) will expose transitive arch-specific modules. Maybe it's resolvable with preprocessor arch detection and things Just Work on Linux? Not sure how this works for macOS universal builds though. |
Can we have a meeting in 2 weeks? Since Clang/LLVM18 will be branched in the end of the next month, I really hope we can achieve this before that. |
I'll send out a scheduling poll to figure out when the best time to meet is either next week or the week after. It would also be good to have some more detailed info for the other suggestions in the thread (what Boris wrote is a good level of detail for a discussion). |
The error towards the end of https://github.com/llvm/llvm-project/blob/main/libcxx/docs/Modules.rst suggests that
If that's really true (and not just for compiled module representation), it's not a good idea to ship it. Before we can do that, we need something that can be used to produce a compiled module for |
That's for the produced BMI file, which is a binary file that is the actual compiled module interface. std.cppm is the source file from which the BMI is produced. |
This is why modules need build system support, they know the user's compilation flags. Then they can create the BMI(s) that match the compiler flags. As @Bigcheese mentioned the source |
I see. Do we need to materialize the source file on disk? Would it be sufficient if the compiler driver had an option to build the |
You would then have to provide a mechanism for detecting when the previously compiled |
Dependency tracking options would already provide that kind of information? Just looking at the file modification time won't cover dependent files even if the source module is materialized on disk.
That probably needs a more complicated handshake between the projects anyway, and a single source file might not deliver all the required ingredients. |
Installs the source files of the experimental libc++ modules. These source files (.cppm) are used by the Clang to build the std and std.compat modules. The design of this patch is based on a discussing in SG-15 on 12.12.2023. (SG-15 is the ISO C++ Tooling study group): - The modules are installed at a location, that is not known to build systems and compilers. - Next to the library there will be a module manifest json file. This json file contains the information to build the module from the libraries sources. This information includes the location where the sources are installed. @ruoso supplied the specification of this json file. - If possible, the compiler has an option to give the location of the module manifest file (#76451). Currently there is no build system support, but it expected to be added in the future. Fixes: #73089
@mordante Should this issues be reopened seeing that the commit in question has been reverted? Also, I see that Clang 18 rc1 has been published which makes me wonder if the ship has sailed on having this functionality in 18. Or are you still planning/hoping to get it in before the final release? |
I don't think we should reopen this since the revered patch is in the clang part. And the job of the libc++ part, to provide option to install it, is completed. |
This feels like splitting hairs to me since there is little use of the installed |
Installs the source files of the experimental libc++ modules. These source files (.cppm) are used by the Clang to build the std and std.compat modules. The design of this patch is based on a discussing in SG-15 on 12.12.2023. (SG-15 is the ISO C++ Tooling study group): - The modules are installed at a location, that is not known to build systems and compilers. - Next to the library there will be a module manifest json file. This json file contains the information to build the module from the libraries sources. This information includes the location where the sources are installed. @ruoso supplied the specification of this json file. - If possible, the compiler has an option to give the location of the module manifest file (llvm/llvm-project#76451). Currently there is no build system support, but it expected to be added in the future. Fixes: llvm/llvm-project#73089 NOKEYCHECK=True GitOrigin-RevId: 8b47bb657b5905d954b9041415020358802407d5
I am trying to understand (and document) what ended up making it into Clang 18 specifically looking at Clang 18.1.5 in Debian (the latest release at the time of writing). The
This seems to match #85637, which is an attempt to backport this to 18 that was abandoned. However, the JSON file and the
Is this understanding accurate or am I missing something? If it is accurate, is it also accurate to assume that the full support for this will only be available in Clang 19? |
Yes, I think this is accurate. BTW, maybe it is helpful to mention that, whether or not the json file gets installed depends on vendors choices. |
Another data point: I've installed Clang 19 snapshot from Debian experimental:
In this version the
Even though the JSON file and the |
Oh, this should be a bug IIRC. @mordante. Maybe we didn't receive bug reports just because no one uses it : ( Can you compile a hello world example with |
The |
Yes. I can even do one better and compile it with
Hm, yes, this works for me also with Clang 19 (but not 18). Now I am wondering what is the difference/purpose of |
I tested it with a custom-built 18.1.2, so why 18 doesn't work in your case is an interesting question… |
Personally I think the intention should be |
The manifest is indeed not installed by default in Clang 18. It will be installed by default in Clang 19. This still requires Linux distributions to add these files to their packages or provide additional packages. The intention was indeed to use |
I also checked Clang in 18.1.1 that comes in Fedora 40:
|
Based on |
Right, we seems to have a full spectrum of ways in which this is not working. FWIW, we are shipping the hack in |
I haven't followed the whole thread here, but our intention was for this to work in LLVM 19 out of the box. I believe we don't expect anything to work well with the 18 release. Libc++ should be fulfilling its part of the contract here, we do install the module files by default. That still requires the distributions to actually install those files as well. If Clang misbehaves when the module files are installed properly, then that would be a Clang bug. |
@llvm/issue-subscribers-clang-modules Author: Boris Kolpackov (boris-kolpackov)
I've been trying out the `std` module in `libc++` with `build2` (per the [Modules in libc++](https://github.com/llvm/llvm-project/blob/main/libcxx/docs/Modules.rst) documentation) and everything seems to work without any issues, at least on simple examples. Very impressive, I must say.
There is just one snag: > Once libc++'s implementation is more mature we will reach out to build system vendors, with the goal that building the BMI files is done by the build system. In I can see two immediate questions about installing Regarding where to install, I would suggest just installing it next to the headers (e.g., into
The only objection I've heard to installing modules next to headers is that "it's not a header" and "it may be included by mistake". While both are true, we are already installing non-header files (e.g., inline/template implementation files) and nobody seems to be including them by mistake. We could also take an extra measure to preclude this by, for example, installing the modules into a subdirectory (e.g., Regarding making Thoughts? P.S. For completeness, let me also mention what I've done as a (hopefully) temporary measure in @mordante @iains @ChuanqiXu9 @dwblaikie @mathstuf |
Let me try to give a summary since it is too long and I am not sure if I understand correctly.
Also in the case |
@ChuanqiXu9 I personally still don't understand whether it is
You seem to suggest that it's the latter (actually, you use one option in the first sentence, and the other in the second, so I am not really sure what you are suggesting, if anything). I upgraded to the latest Clang/libc++ 19 pre-release from Debian experimental:
The modules are installed:
The
The
So let me repeat my earlier request: Can someone give an authoritative answer as to what mechanism a build system vendor should use? We are almost 60 comments down in this issue, and we still can't seem to figure this out! |
We said, it should be |
@ChuanqiXu9 Ok, thanks for the clarification. Could you also explain what would be the incentive for me as a build system vendor to later switch from |
To be honest, for my memory, |
I've been trying out the
std
module inlibc++
withbuild2
(per the Modules in libc++ documentation) and everything seems to work without any issues, at least on simple examples. Very impressive, I must say.There is just one snag:
std.cppm
is not installed. Quoting the relevant part from the above documentation:In
build2
we have the ability to build BMIs of external libraries on the fly (this is needed for consuming modules from installed libraries). So ifstd.cppm
were installed somewhere where we could find it, then this would work pretty much out of the box in our case.I can see two immediate questions about installing
std.cppm
: where to install it and how to make it self-contained (currently, it#include
s a large number of files from thestd/
subdirectory).Regarding where to install, I would suggest just installing it next to the headers (e.g., into
/usr/include/c++/v1/
). I can think of two advantage with doing it this way:There is already a way for a build system to find this directory (by parsing the header search paths in the
-v
output) and some build systems already do this (for example, inbuild2
we extract the compiler's header and library search paths).The
std.cppm
file will end up in the right distribution package (e.g.,-dev
on Debian,-devel
on Fedora, etc) without any extra effort.The only objection I've heard to installing modules next to headers is that "it's not a header" and "it may be included by mistake". While both are true, we are already installing non-header files (e.g., inline/template implementation files) and nobody seems to be including them by mistake. We could also take an extra measure to preclude this by, for example, installing the modules into a subdirectory (e.g.,
/usr/include/c++/v1/modules/
).Regarding making
std.cppm
self-contained, I assume there is no desire to install all the 100+ files fromstd/
that it includes (though there is nothing technically bad about doing it, just feels unnecessary). It seems the easiest would be to just append their contents tostd.cppm.in
instead of#include
ing them. Though I am not sure if this is easy to achieve in CMake.Thoughts?
P.S. For completeness, let me also mention what I've done as a (hopefully) temporary measure in
build2
to supportimport std
right now. I've made a self-containedstd.cppm
(as described above), patched it slightly so that it works for bothlibc++
17 and 18, and bundled it withbuild2
. In other words, instead of being installed withlibc++
, it is being installed withbuild2
(again, hopefully temporarily and intobuild2
's own directory, naturally).@mordante @iains @ChuanqiXu9 @dwblaikie @mathstuf
The text was updated successfully, but these errors were encountered: