Dependency Handling

[ramcdona]

In my program, I generally follow the CMake 'SuperProject' pattern for my dependencies. I believe it offers some advantages over the current approach for Manifold.

Can you assess the feasibility in moving to this kind of system. If it is feasible, are you willing to accept patches in that direction?

First, a little background....

For those unfamiliar with it, in the SuperProject pattern, a project is organized into a Libraries project and the Application project. Typically, the Libraries project contains outside dependencies using CMake's ExternalProject_Add capability. The Libraries project has per-library variables USE_SYSTEM_FOO. When false, the bundled library is compiled. When true, it is skipped.

The Application then uses CMake's find_package( FOO ) mechanism to find a library and set appropriate variables pointing to include files and compiled library files. You can point find_package at the Libraries project, or it can search your system (good for shared libraries on Linux systems), or you can point it at a version of the library in your own filesystem (good for developing the library and application in parallel).

ExternalProject_Add can pull projects down via git -- or you can bundle *.zip files locally. ExternalProject_Add also provides a mechanism to use a different build system for a Library than for the main project. In case you need to use clang for something and gcc for the rest.

This approach is called a SuperProject because their is a third project (SuperProject) that packages the Libraries and Application into one. That way, someone 'just' compiling a program, (but not really developing it) can use the SP as a one-shot build process.

Here are the perceived problems I see that I think could be remedied with a SuperProject approach...

• Manifold embeds some libraries that my project already embeds. Clipper2 (I'm still on Clipper1, but with intent to upgrade), and GLM. If Manifold used the SuperProject approach, I could tell it to use the system provided library and point it to the one I have already bundled.

• My program needs to be able to build on computers with no graphics capability -- where the system has no graphics libraries installed, and graphical programs are not allowed. In my program, GLM is only used on the graphics side of things, so it can be disabled with a CMake flag and GLM is no longer required by the program. It may be that GLM just does math and doesn't actually require OpenGL or any other graphics libraries -- so perhaps this is moot, but if GLM is only needed by unit tests or demo programs, then I would encourage you to make the dependency dependent on the same CMake variable as controls building the tests and example programs.

• Similarly, if I turn off the parallel build, it would be great if I didn't need to build the parallel library (thrust). Even better if I don't even have to download or have it in the first place. Thrust has its own dependencies (cub and libcudacxx, or TBB or OpenMP).

• My program needs to be able to be built on computers with no connection to the internet. This means no build instructions that involve 'git clone', no submodules, and no ExternalProject_Add that points at a VCS. I need to be able to point them at a single *.zip file to download -- and that single file can be transferred to a computer without connectivity, where the program can be built (if required).

[ramcdona]

I took another look at GLM -- although it has GL in the name, it explicitly does not depend on <GL/gl.h>, <GL/glcorearb.h>,
<GLES3/gl3.h>, <GL/glu.h>, etc.

So my 2nd bullet above is moot. It is still true, but manifold has no problems passing that criteria because of GLM.

[elalish]

We are 100% ready to accept patches to improve our builds. I am very much not a CMake expert, and the main reason our builds work as well as they do is from excellent contributions from devs who know more about it than I do. For my part, I generally evaluate new approaches by how well they run on our Github Actions CI and what our others users say about their ease of use.

@pca006132 knows more about our CMake than anyone at this point, so I trust his opinion. @kintel and @t-paul also have opinions on how to make Manifold easier to consume for OpenSCAD, and I would guess their desires largely fall in line with yours, but I'll let them use their own words.

I think this is mostly about points 1 and 4. As you say, 2 is moot, and 3 we're already working on (removing Thrust) but we're not quite there yet and it currently supplies some algorithms we use even in single-threaded mode.

[pca006132]

I somehow forgot to answer this one.

My program needs to be able to be built on computers with no connection to the internet. This means no build instructions that involve 'git clone', no submodules, and no ExternalProject_Add that points at a VCS. I need to be able to point them at a single *.zip file to download -- and that single file can be transferred to a computer without connectivity, where the program can be built (if required).

We are already doing something similar with tbb, but we use FetchContent instead of ExternalProject_Add. We can use this for other third-party dependencies as well. You can specify a path that points to the third-party library when you configure cmake, e.g. -DFETCHCONTENT_SOURCE_DIR_GOOGLETEST=$(GOOGLETEST_DIR), so no internet connection is required. We can also let it use the system provided package if available.

I wonder what is the benefit of using ExternalProject_Add comparing with FetchContent, I am not very familiar with this. Can ExternalProject_Add override the default download behavior or will users have to change the CMakeLists file?

[ramcdona]

I don't remember if git submodule was an option when I started doing it this way.

I name my zip files like... manifold-5e103931f119.zip Where I use the first 12 characters of the git SHA that corresponds to the exact version of the dependency I am using.

I can download these zip files from GitHub (with the SHA in the name already). I just navigate to the branch I want, click 'Browse Files' and then download a zip from the Code pull-down button. When you download from GitHub in this way, it will not recurse submodules for you.

So, to get submodules, you have to clone a repo locally (recursively) and then build a zip.

It may seem small, but for non-experimental dependencies (those that I don't have to hack on), I prefer to not have a local git repo and I like to just download the *.zip from the GitHub site. My application has about 15 libraries that I include using ExternalProject_Add, and while there are four or five that I hack on occasionally (to fix issues that arise in my application), there are about ten that I have never touched.

[pca006132]

I don't understand the benefit of this comparing with using submodule, I think this is just a matter of personal preference, and from what I observed people generally prefer submodules over storing binary files in git (of cause using system library would be the best).

[ramcdona]

If you use submodules, you need internet access to recurse into all the submodules.

My users who build on computers that do not have internet connections typically go the GitHub page and download a *.zip of my project. Doing that will not recurse submodules.

I want them to be able to download everything they need in one file. Likewise, when I embed a dependency, I want to get everything in one file.

I agree that it is largely a matter of personal preference. I also agree that most projects try to avoid large binary files in a repository.

My need to compile on an air-gapped machine -- where the user only brings in a single *.zip file -- is where most of this comes from.

[pca006132]

I wonder what other users think about this. @t-paul @stephomi

I think the main problem is thrust, thrust is a relatively large dependency which itself has submodules to cuda library. A zip file containing thrust (and its dependencies) is already over 69MB, which is larger than our repository already.

For me, I think it would be better to provide a release archive to users for them to compile on air-gapped machines. Alternatively, when your project include us as a dependency, you can also recursively clone us (or download the FetchContent sources and point to them using cmake arguments) and store in a zip file.

[ramcdona]

That is fine with me. I appreciate your help.

I will also be able to recurse selectively -- for example, I can skip GLM because I already have it in my package, but recurse something else if needed. Or, I can recurse none of it and provide them as system provided libraries to Manifold.

The main things that will help me is to make building the internal libraries optional (and a separate option from whether they are used or not). And then using FindPackage(FOO) consistently such that Manifold's build system doesn't care whether it is using an internal or external library.

I guess the final point is to then make Manifold's build ready to be used via FindPackage( Manifold ). The old way (Module mode) would be to write a FindManifold.cmake script -- the 'Modern CMake' way (config mode) is to generate a Manifold.config file that sets all the variables appropriately. The important part for a user is documenting how it works and what variables are set i.e. how to then use Manifold in an application.

[stephomi]

Prebuilt binaries is obviously a no go for me (mobile, wasm, debugging, etc).
It’s fine to have them in the release page, for example https://github.com/embree/embree/releases/tag/v4.2.0

One thing a bit annoying is that manifold pulls gtest and nanobind which aren’t useful for standard build.
I’ve never used FetchContent or ExternalProject_Add, so I don’t know if these are alternative to submodules (assuming CMake is the single build system used on the project).

[pca006132]

Revisiting this, we do use FetchContent for some dependencies but not all. I view FetchContent as a fallback to distro provided packages, but not sure if this is the right view. We can put nanobind in FetchContent as well.

[ramcdona]

From my point of view, the important thing (to enable use of system installed libraries) is more about how the libraries are referenced by other parts of the CMake system (vs. how the libraries themselves are built).

Case 1) Library is discovered by FindPackage( foo ). This sets variables such as FOO_INCLUDE_DIRS and FOO_LIBRARIES which are then used where needed in CMake.

Case 2) Library is included in project by AddSubdirectory( foo ).

1. Allows easy switching between a) bundled library, b) system installed library, and c) development library installed in a user's directory. This is done by setting variables that act as hints to FindPackage(). Everything in CMake world works as-is no matter the library's origin.

2. I don't know how to point AddSubdirectory() at a system installed library instead of the thing included in the package (whether via submodule, simple inclusion, or FetchContent).

In my experience, ExternalProject_Add() is the approach that best supports 1).

[pca006132]

We do have users using add_subDirectory, I think we need better tests for case 1, sadly I am not very familiar with cmake in this regard. I think there are things like fancy export target and such that allows you use something like manifold::sdf as a target in target_link_libraries and setup linking and include automatically. Not sure how to use this.

[ramcdona]

Yes, the fancy stuff is generically called "Modern CMake". It has been evolving since 3.0.

Modern calls to FindPackage() will populate targets (instead of setting old-fashioned variables).

If the sub-project being added uses targets, then I think that add_subdirectory will expose them in a very similar manner as would FindPackage() of it -- but I'm less confident of it than I am the other way.

Even if this is true, you need to replace an add_subdirectory() with a FindPackage() in order to use a system-installed dependency. I prefer to use FindPackage() in both cases.