C/C++ Backend

[sureshjoshi]

C/C++ Backend

Preamble

There is no explicit proposal below.

This is intended to be a bit of a brain dump and request for ideas/opinions/thoughts on creating a compilation and packaging workflow for the C and C++ backend(s). At the moment, there exists the recently added cc_sources target and ClangFormat plugin.

My hope is that we can come up with some ideas on each of the following topics, to provide a path forward on each high-level subject and incrementally build out the backend(s).

While the CC backend is in experimental, I think two reasonable user/project-types to cater towards are:

• New C/C++ projects in a monorepo (extra points for a multi-language monorepo)
  • These users may be looking for something simpler than manually creating Makefiles, or tediously copy/pasting CMakeFiles and tweaking
• Existing projects in a monorepo (or being moved into a monorepo)
  • These users may not want to gut the existing build infrastructure immediately, but would still want to leverage a multi-language, monorepo build tool
  • There should exist a migration workflow to Pants, but it needs to allow for a gradual, continuous process (no hard boundaries)

Conveniently, I happen to be in both of these camps... What are the odds? :)

There exists another category of projects, with users actively looking for a new build solution to replace theirs - but these tend to be rare, and usually live in the edge-cases with what they require (eclectic compilers, super-massive repos, less common CPU architectures, etc...). I wouldn't recommend spending much time targeting these workflows, even though, I again happen to be in this category for one isolated project.

Compilers/Toolchains:

• How are they installed/called?

  • Can they be easily downloaded (like ARM)?
  • Or are they expected to be installed by a package manager (like gcc)
  • Or maybe a mix of both (like clang/llvm)
  • We may need to support downloaded and system-level toolchains

• Should they be specified per target or globally?

  • Generally a single compiler should be used for related code and dependencies
  • A mono-repo could compile different targets for different architectures
  • For embedded work, the target-architecture may differ from the host-architecture (where unit tests can be run)
  • There may be different compilers/toolchains per OS

Package Management

Package management is a hot sloppy mess in C/C++ land.

• single file headers

• git subrepos/subtrees

• conan

• vcpkg

• pmm

• buckaroo

• ...

• What is the recommend approach for projects that use 3rd party PM tools?

  • Would we require plugins for package management?
  • Would we support the biggest (conan and vcpkg right now I think)?
  • Would the user need to handle this on their own?

Linters/Formatters:

• At the moment, clang-format is retrieved via a 3rd-party-generated Python package - ideally this would be retrieved from a 1st-party location
• Several common linters are clang/llvm plugins, which means that even if the chosen compiler is gcc, clang/llvm may need to be downloaded regardless

Unit Testing

While not as bad as package management, there are several oft-used testing tools (excluding mocking tools):

• Catch

• Google Test

• Boost.Test

• CppUnit

• CUnit

• Do we need to support selecting a test runner as a plugin?

Compilation/Linking flags and options

There should be an option to set these types of flags globally, and then override them per target.

• C standard (89, 99, 11, ...)
• C++ standard (11, 14, 17, 20, ...)
• C++ extensions (Use -std=c++11 vs -std=g++11)
• Compilation flags (e.g. -Werror)
• Defines (e.g. -DUNIT_TESTING)
• Static vs dynamic linking
• Debug/Release
• Optimization levels
• ...

Build Generators

Is it worth considering exporting to Makefiles or .ninja files (ala CMake)?

Potential next step?

Create a CMake plugin to help with the "existing project" workflow as described above. Aiming for a "just works" solution for the user, with minimal BUILD intervention.

CMake plugin would essentially call down into cmake/ctest/cpack/etc, pass along some configuration items, and define the destination directory for the install command. Not very Pants-y internal to the C/C++ code, however, it would allow monorepo integration (like an outside-in approach to development).

Target Types (WIP)

How are transitive dependencies handled? Do the settings flow through, or do we need target-limited equivalents
How do private/public headers fit in?
How are public headers published (in the digest?)?
How do single-file headers or pre-compiled sources fit in?
Option de-duplication?

cc_library

? compile options, linker options, static/shared link, source files, libraries, headers

• dependencies
• compile_options - string list of compile options, added to the existing toolchain options
• link_options - string list of linker options, added to the existing toolchain options
• defines - string list of defines to add to compilation, prepended with (-D) unless it is already there

cc_binary

? compile options, linker options, source files, libraries, headers

• dependencies
• compile_options - string list of compile options, added to the existing toolchain options
• link_options - string list of linker options, added to the existing toolchain options
• defines - string list of defines to add to compilation, prepended with (-D) unless it is already there

cc_test

References

Adding a New Language to Pants

[Eric-Arellano]

For third-party package management....maybe we use a distinct backend per ecosystem that you explicitly opt into? pants.backend.cc.third_party.conan

I've never done third-party stuff w/ C/C++, would you ever mix package managers in a project? If not, can you envision having multiple projects in a repo, each w/ a different package manager?

--

For test runners, I at first was going to suggest a similar idea. Have a c_unit_test vs cpp_unit_test. You opt into the backend you want.

Buf oof that's a lot of targets, especially when you add target generators. I suspect the metadata is similar with all the test runners, like a timeout field? And we can use Plugin Fields to add custom metadata we need.

I'm not sure how each test runner knows whether to run on a cc_test target though. Possibly we have a field where the user tells us which test runner? We let them set a repo default?

--

For compiler/toolchains, would something like JVM and Python's resolves make sense? https://www.pantsbuild.org/v2.11/docs/python-third-party-dependencies#multiple-lockfiles

[sureshjoshi]

I've never done third-party stuff w/ C/C++, would you ever mix package managers in a project? If not, can you envision having multiple projects in a repo, each w/ a different package manager?

I'm not going to say no... But, I would be surprised to see that. I typically just build from source, but I think typically people who use them fall into their preferred "camp". The only time I would envision it is during the migration of a repo, not a steady-state one.

I'm not sure how each test runner knows whether to run on a cc_test target though. Possibly we have a field where the user tells us which test runner? We let them set a repo default?

The idea I had originally was a generic cc_tests (or c_tests + cpp_tests), and the test plugin uses that target, but applies its own set of dependencies/meta/object files/whatever. There is no real convention for test files, and it's all just about how they're written and what includes/libs you send in with them.

Again, don't think there should be multiple test backends, except while migrating. This is more likely than multiple package managers, as I (for example) have migrated a project from Google Test to Catch, and kept both alive and running during that migration (since the migration was file-by-file).

I think the general Pants idea is to create new targets as applicable for a plugin, but are there examples where "stable" targets have a field allowing plugin-based customization?

For compiler/toolchains, would something like JVM and Python's resolves make sense https://www.pantsbuild.org/v2.11/docs/python-third-party-dependencies#multiple-lockfiles

I have to look into this - I didn't even think about going this route

[Eric-Arellano]

I think the general Pants idea is to create new targets as applicable for a plugin, but are there examples where "stable" targets have a field allowing plugin-based customization?

Not that I can think of. Plenty of examples of adding a new plugin field. But I don't know of a field that chooses which test runner to go with, especially where the list of possible runners is not known in advance.

[siemato]

I recently came along a post on hackernews about Uber using Zig's compiler tools to build their C/C++. Reasons are Zig's cross compilation story, having a linker on their own. Zig is known to improve cross compilation with Go and Rust FFI as well. Added the adoption and support from Uber (support contract with Zig foundation) along with another big tech does secure long term support.

Maybe this is a compelling reason to favor Zig over Clang/GCC. Any opinions on that?

[sureshjoshi]

You raise some great points! I don't know much about Zig, as it's never come up in my use cases.

I'm not sure if I made it clear (and if I haven't, I have some editing to do), but the goal is to be toolchain independent. As in, the compiler toolchain is essentially a plugin (probably living in the pants main repo), but definitely not forcing one or the other. I personally use either/both gcc and clang depending on the project and compatibility, so locking into one or the other isn't a valid option.

So, that would mean a Zig (or other) toolchain could be dropped in as well.

In my upcoming experiments/spikes, I'll be using those, but that's entirely due to my familiarity - not because they will be the sole options.

[siemato]

As far as I undestand, Zig has got 2 toolchains, one for development purposes without optimizations and one for release builds. release builds use LLVM. that is the one I would go for.

This is the corresponding blog post and the repository (for Bazel)

To sum up the reasons: 1. Zig is a hermetic toolchain to begin with, 2. You can pass in your preferred libc as an argument (glib/musl/whatever), 3. It integrates well into cgo builds (or python/rust ffi) and 4. cross compilation for linux/darwin is easy because host and target are independent - it's just stating a different target.

sounds compelling, especially the hermetic build argument. The question is: how long does it take to get familiar with the toolchain? Can we use the Bazel toolchain as a template?

[sureshjoshi]

Thanks for the link! That post definitely makes me want to consider Zig for my first implementation of the toolchains, since it appears to save a ton of grief vs what I've already done with clang.

We could possibly look at Bazel for this, but I wouldn't want to get bogged down too heavily in their ecosystem with this. I've used Bazel for cpp stuff in the past, and I honestly just switched back to CMakeFiles. Not a knock on Bazel by any means, but it just wasn't worth the mental model cost.

Pants has a large language selection already, so the goal would be to do things "the Pants way", as I've started adding other language plugins - and it's pretty straight-forward. I think C/C++ is harder because it's a suite of tools, and infinitely customizable - so I think a lot of the difficulty will be in handling customizability, rather than just getting compilation working (which I already have done, for super trivial stuff).

[siemato]

We have no time pressure for now, right? Can we keep an eye on this and decide a bit later what to do? I'll be able to step in during the second half of the year as well. Maybe it's something we can target for Q3/Q4.

[sureshjoshi]

Yep, no time pressure at all - I'm just working on a 6 month project which has C++/Python/etc - and I figured it would be a good time to roll something out :)

[sureshjoshi]

Throwing an idea into the void... For "CMake-support" - instead of calling into cmake, maybe this plugin would enhance the Tailor command to auto-generate a reasonable equivalent? Maybe even both, anyways...

It's hard (to the point of not trying) to parse any/every CMakeLists file into a BUILD, but the 80/20 approach feels like a layup. Get the sources all in, create libraries/executables, a bunch of compiler options - and stop there.

[siemato]

I would support that, but I would like to get the backend running first. Actually, I would consider it a meaningful contribution to stabilizing the toolchain, because we could test it with meaningfully large projects.

Edit: Thinking further, we need to provide a great onboarding experience. DX should be a reason to bring existing projects over to Pants, not just starting new ones. As such, a proper language toolchain should implement everything required to bring projects over without reworking and breaking too much things. This is a larger topic than just "Let's port over CMakeFiles".

Edit2: Let's defer that one until the toolchain works, then try to improve the onboarding for projects.

[mbodmer]

I am also very interested in a possible C/C++ backend, hence my thoughts below.

I suggest it would make sense to integrate CMake/CTest/CPack (priority in this order) without any package managers first. Package Managers in the C/C++ are mostly inexistent (no common understanding or standards) and probably Pants should (partially) take the role of a PM, at least for dependency resolution.

CMake Build support

Pants should provide a way to configure CMake and it's possible Options (-Dxxx) or maybe more flexible via the new CMake preset mechanism: https://cmake.org/cmake/help/latest/manual/cmake-presets.7.html

Dependency resolution

For a CMake project to find it's dependencies there are sadly too many mechanisms as there is a long legacy. The most often used are however the CMake native Config Mode: https://cmake.org/cmake/help/latest/command/find_package.html using *Config.cmake files and also pkg-config *.pc files: https://cmake.org/cmake/help/latest/module/FindPkgConfig.html?highlight=pkg%20config#module:FindPkgConfig. They are created by the CMake build in the "install" target. The install place however has to be managed and injected by Pants, best handled by the DESTDIR environment variable: DESTDIR

Toolchain/SDK

On the toolchain side the most generic way I could think of is along the way Yocto and also e.g. GNU does it, is by defining a build environment by setting ENV variables: CC/CXX/CPP/..., and everything else needed for a specific toolchain. Often SDKs have environment setup scripts which can be sourced prior to building.

I am not sure how to handle these, but somehow I think Pants should just require the correct build environment to be set prior to execute Pants and thus respect it. This however only works if the complete build is for a single target w/o building native tools. Otherwise Pants has to set the correct environment and needs a mechanism for this. Probably also best using these environment setup scripts provided by the SDK.

Future integration/introspection

The described CMake integration above is by creating the respective "goals" on the commandline.
Maybe in future when Pants needs tighter integration and more knowledge about the system to build, maybe it can integrate with CMake Server: https://cmake.org/cmake/help/v3.7/manual/cmake-server.7.html

Edit: just found out CMake server mode is removed from new versions, new API ist the file API: https://cmake.org/cmake/help/latest/manual/cmake-file-api.7.html#manual:cmake-file-api(7)

Edit 2: reading the CMake File API docs it seems to me that using the File API would be a natural addition to the commandline integration described above, to get more insight into the CMake Project. That means it wouldn't take another approach, but can be added and thus build on the work already done.

[sureshjoshi]

Great feedback, thanks! I should note that there exists the most barebones CC backend imaginable right now: #16424 - it just needs some more tests and a cleaned up naming convention, before I can PR it. Finally have time this week to finish that.

One attempt I'd like to make that Pants can't currently support is a tailor plugin. So, in addition to "proper" CMake support, I would like to try to have a straight-forward migration path, rather than just Pants running CMake which then turns into Make/Ninja files, which then compiles.

We'll see how it works out in practice.

[siemato]

Yes, the original thought process revolved around a path that leads to a more general applicability, that's where the Zig-Toolchain for C/C++ compilation and the idea of gradually adapting LLVM parts came from. The first enables hermetic builds and splitting up LLVM steps can lead to better artifact caching and implementation of other languages that build on top of LLVM as a frontend.

I originally planned to be active for this but I'm still out of order after a surgery, sorry guys.

[sureshjoshi]

No worries, feel better!

[sureshjoshi]

Closing this as "resolved" for the most part, I think the rest of the discussion will happen with PRs and during the experimental CC phase.