Add Python language binding using libnanobind

[Walter-Reactor]

Maybe not 100% ready for primetime in it's current state, but I wanted to share

Generates a single nanobind.cpp file which can then be compiled, along with the C++ binding layer, to produce viable python bindings.

Needs unit tests, and a more robust readme/packaging solution. It should be possible to output a wheel directory per https://nanobind.readthedocs.io/en/latest/packaging.html, which could be used to construct a proper python wheel via pip

[Manishearth]

Thanks!

Our preference is for Diplomat-generated bindings to not go through an additional tool. In part, the goal is to produce idiomatic APIs with direct control via attributes, and an additional bindings layer doesn't always work well for that. Especially since Diplomat supports things like returning borrowed references which requires nontrivial work to make safe in managed languages.

So I'm not opposed to adding a python-nanobind backend, but if you're still open to doing it differently I'd advise doing something that generates pure python code, modeled off of the JS or Dart backends.

[Walter-Reactor]

Unfortunately outputting .py files cannot produce an optimal result, and the closest thing to it would still require an additional compilation step. Nanobind is the state-of-the-art for producing python plugins, and the next best option would be using the raw python CAPI in a .c file, which is really painful, and would basically require in re-implementing nanobind for good results.
This backend admittedly needs a little more work to produce a friendly packaging story, but I plan on addressing that in a bit.

Longer term, python may wind up with multiple backends for different use-cases. python-nanobind for the performance conscious, python-ctypes or python-cffi-abi/api for people wanting something using raw python.

[Walter-Reactor]

Semi related, is there code anywhere in the tool for pulling out the name of the crate being scanned? I'd like to be able to use the crate name as the default python namespace, with a possible override. I'd been just using the 'namespace' attribute but it's a bit clunky & I don't think it's actually a good solution

[Manishearth]

Semi related, is there code anywhere in the tool for pulling out the name of the crate being scanned?

No. We could do this, but so far the way this is handled is that you have a config file that you pass into the tool. See the Kotlin backend for this.

No opposition to adding such a thing to the Python backend, though.

[Manishearth]

Unfortunately outputting .py files cannot produce an optimal result, and the closest thing to it would still require an additional compilation step. Nanobind is the state-of-the-art for producing python plugins, and the next best option would be using the raw python CAPI in a .c file, which is really painful, and would basically require in re-implementing nanobind for good results.

Alright, fair. It's not great that Python's in that situation, but that's fine. CFFI was what I was thinking of.

The main question then becomes: how does the nanobind backend handle APIs like this:

impl Foo {
   fn borrow_self(&self) -> &Bar {
      &self.bar
   }
}

In the C++ backend you are expected to "hold it right" and not persist the borrow longer than usual, as with any other C++ borrowing API. In JS, Kotlin, and Dart, however, the produced object Bar will contain a reference to its parent Foo so the gc doesn't delete things. Would this be possible with nanobind?

If not, should we have a supports key for this and disallow such APIs with nanobind? (Worth doing even if this is theoretically-possible-practically-not-prioritized)

[Walter-Reactor]

Totally possible, yeah. Nanobind is meant to be really performant for this kind of thing.

For functions with this kind of lifetime annotation, the optimal nanobind would look like
.def("borrow_self", &Foo::borrow_self, nb::rv_policy::reference_internal);

The code in this PR doesn't have the logic for rv_policy setting, so would use the default which would cause a copy of the returned value to be created in python-land. However, if we marked it with the getter attribute, it'd instead use .def_prop_*, which uses the reference_internal policy correctly.

If you're interested, there's a really nice section in the nanobind docs on object ownership that covers all the relevant stuff: https://nanobind.readthedocs.io/en/latest/ownership.html

[Walter-Reactor]

Semi related, is there code anywhere in the tool for pulling out the name of the crate being scanned?

No. We could do this, but so far the way this is handled is that you have a config file that you pass into the tool. See the Kotlin backend for this.

No opposition to adding such a thing to the Python backend, though.

Gotcha. That's actually probably a little cleaner since it means the name of the python package doesn't have to match the crate name. Out of curiosity, why a config file rather than a top level attribute?

[Manishearth]

Could be a top level attribute, we don't have infrastructure for parsing those. Would honestly love a way to just specify config keys at the top level, and an improved, unified config system.