feat: support recursive stubgen

[cemlyn007]

To support producing stubs for submodules.

#43

[nicholasjng]

Thanks for taking this up! Here are a few comments.

If I may ask: In which setup do you use recursive stubgen? I'm wondering if for multiple extensions, you need to make all extension labels available to the stubgen binary as data, cf. L198 in the diff.

[nicholasjng]

I think we might need some logic to ensure that -o <filename> (the output file, which is optional) and -r do not appear in the same stubgen call.

Could you try if something like this would work:

if recursive and output_file is not None:
    fail("Cannot specify an output file if recursive stubgen is requested")

Alternatively, we could let the stubgen script itself throw the exception, but then we should make a note of this in the docstring. Do you have a preference?

[cemlyn007]

Personally don't mind, I'll add the check in the Bazel side

[nicholasjng]

This is to work around the recursive + output_file case, right? We should catch that earlier, see my above comment.

[nicholasjng]

This is potentially very expensive if many packages appear under runfiles, and if they have stubs in them, they might be erroneously copied into $(BINDIR).

I wonder if there is an option for stubgen to put all recursively generated stubs into the same directory? Maybe -O $(BINDIR)?

[cemlyn007]

I can't believe I didn't see that option XD, yeah I'll get it done! Thank you so much

[cemlyn007]

Thanks for taking this up! Here are a few comments.

If I may ask: In which setup do you use recursive stubgen? I'm wondering if for multiple extensions, you need to make all extension labels available to the stubgen binary as data, cf. L198 in the diff.

I started a project based off the Nanobind examples bazel branch, but my code was getting a bit chunky so I was looking to modularise the code but I still only wanted a single Nanobind extension. I saw in the Nanobind documentation they have support for sub-modules.

I was hoping ultimately have source code looking something like:

src/packageA/moduleA_0/<some_python_files>
src/packageA/ext_moduleA_0/ext_moduleA_0.cpp <- The Nanobind extension
src/packageA/ext_moduleA_0/sub_module_A_0_a/<some_cpp_files_included_by_ext_moduleA_0.cpp>
src/BUILD.bazel

And produce a wheel that looks something like:

packageA/moduleA_0/<some_python_files>
packageA/moduleA_0/ext_moduleA_0.so
packageA/moduleA_0/__init__.pyi
packageA/moduleA_0/sub_module_A_0_a/__init__.py

But my MR is not entirely correct with naming the stubs correctly, at the moment packageA/moduleA_0/__init__.pyi looks like packageA/moduleA_0/_main.src.__init__.pyi (I think but AWOL from my home desktop), and they're not being picked into the wheel, although that might be resolved when I try your comment with using the -O flag in the stubgen script 🤷🏻‍♂️

[cemlyn007]

I had another go, I also have modified the Bazel Nanobind example which can be found here for a minimal example. Unfortunately the root stub is named like _main.src.pyi which led me to do some awful hackery, would appreciate any advise on how to better tackle it!

If you use this branch with the this example branch, you should get a working (brittle) example.

[nicholasjng]

I remember encountering this when attempting to write stubs relative to the working directory in a Bazel project - I think you cannot avoid botched stub names when specifying the output directory and it's not the same as the current working directory.

What I would like is this:

src/nanobind_example/__init__.py
src/nanobind_example/nanobind_example_ext.pyi
src/nanobind_example/sub_ext/__init__.py
src/nanobind_example/sub_ext/__init__.pyi # <- this would contain stubs for sub()

I'll still have to check what stubs are actually produced with your changes, but tl,dr: I think the correct solution is that top-level module extensions get named stubs, and submodules get __init__.pyi files.

Maybe we can avoid the move hack by tweaking some logic in nanobind.stubgen, I'll check it out.

[nicholasjng]

Could you explain this comment? Since the output dir is relative to BINDIR, it cannot be absolute.