[Question] How to add a build C extension to a python module

[D4N]

I have a mixed C++ & Python project, where a part of the C++ API is exposed via boost.python (this part works like a charm, thank you meson-devs!). I have however now the problem, that my C++ extension ends up somewhere in build/src/extname.cpython-36m-x86_64-linux-gnu.so. The python code on the other hand stays in the source directory (under lets say py/).

Is there a canonical way, how to allow the C++ extension extname.*.so to be installable via distutils? I know that I can provide a MANIFEST.in file to setup.py to find the extension, but I can't put that back into the source directory with meson (as that violates the out of source build idea) for setup.py to find it.

Bonus question: I have written the documentation with sphinx, which also documents the python code and requires that I can import it (also the C++ extension). Imho the easiest would be to run setup.py install --user before building the documentation (but that feels wrong as the install step is run before the build doc step plus it pollutes the user's directory just for building the documentation). Is there a better way? Maybe copy the whole python module into build/ and export PYTHONPATH (although that is only possible via hacks for run_target), if that is somehow possible?

[astavale]

From what I know the Meson approach is to integrate some custom scripts in to your meson.build file.

I've done this with Vala, but hopefully you'll be able to apply the pattern for Python. So Vala has the valadoc tool for generating documentation. The following generates the documentation in the build directory:

valadoc = find_program( get_option( 'valadoc_path' ) )

custom_target ( 'docs',
                input: get_option( 'valadoc_input_file' ),
                output: 'docs',
                command: [ valadoc,
                  '--force',
                  '--pkg=posix',
                  '--package-name=blkid',
                  '--directory=@OUTPUT@',
                  '@INPUT@'
                  ]
              )

This creates a docs target.

As far as installing goes take a look at Installing and the meson object. You may have noticed a lot of GNOME projects use a meson-postinstall.py script that is added with meson.add_install_script('meson-postinstall.py').

[dcbaker]

I think the python module has the helpers you want: https://mesonbuild.com/Python-module.html#extension_module

[jpakkane]

You might want to take a look at how Meson's module building unit test does it.

For distutils I don't have an answer. However I do know that there is a way to tell distutils to invoke external build systems. This is how e.g. projects using SCons get turned into Python packages. I have never done it myself, though.

[D4N]

Hi folks, thanks for your suggestions! I haven't cracked the setup.py integration yet, but I have more or less managed to integrate the documentation build into meson.

The main problem here is twofold:

1. Sphinx really does not want to perform builds with the configuration file outside of the source tree. This requires that one has to lie to meson and pretend that Doxygen generates its output into the build directory although it creates it in the source tree. Fortunately, this works quite well. Also, since this is sphinx' fault, there's nothing meson can do about that.
2. Setting PYTHONPATH for custom_target doesn't seem to be supported via an env option, I have used the workaround from Add env: argument to custom_target, run_target, run_command #2723. Adding this option to meson would be helpful but is currently not a show stopper, as the workaround works fine.

Once I have found out how to implement a setup.py, I'll report back.

Thanks again for your suggestions and your work on meson! It's a great build system, quite simple, has a nice syntax and it doesn't tempt you to do crazy specialized stuff.

[Nic30]

Hello,

is there an example how to build a c python extension by meson from setup.py?
I would like to use meson because of dependencies of my library however the pip/PyPi compatibility needs to be sustained.

(And I do not know how to use meson in pip packages.)

[D4N]

@Nic30 I don't know if there is an example, but it shouldn't be too hard to build a Python extension from setup.py. Provided that you can call meson via subprocess.call, I'd suggest to execute the normal build, but instead of via a shell, do it from python. Note that I've never tried that, so there might be some unforeseen issue.

[Nic30]

@D4N I am not afraid of build I am afraid of correct installation on all platforms.
If I find some example I will share the link otherwise hdlConvertor lib. will use meson+pypi in future (~1m from now).

[D4N]

@Nic30 You could take a look at the python test cases that Meson uses: https://github.com/mesonbuild/meson/tree/master/test%20cases/python3, maybe one of them roughly fits your use case? If yes, you could probably just call meson and ninja from setup.py (plus maybe some additional fiddling with the installation path).

[eli-schwartz]

Distributing the built extension via distutils is best solved by not using distutils at all, but using https://pypi.org/project/mesonpep517/ instead.

For generating sphinx documentation, run_target or custom_target have since been updated to accept an env kwarg which sets the PYTHONPATH. Sphinx refusing to run in a directory that doesn't have the conf file, might be solvable by using configure_file(..., copy: true) to copy over the conf file into the build dir?

Is this all sufficient for your needs?

[Nic30]

@eli-schwartz
Yes from doc mesonpep517 looks exactly as thing I need. (to have pip-installable package which will build C/C++ part of package using meson)

Thank you.