Protocol definition with language-specific idiomatic bindings

[kajacx]

This is a feature request to add wit-like protocol definition, from which language-specific bindings with correct function names and signatures can be generated.

Wit comparison

Let's consider a simple example: the protocol will export a function called "calculate", and the host will provide functions "get_x" and "get_y" that the plugin can call during the calculation.

The wit protocol would look something like this:

package example:protocol

world my-world {
  import get-x: func() -> float64
  import get-y: func() -> float64

  export run: calculate() -> float64
}

Plugin side

Currently when writing a Rust plugin, we have to manually import the "get_x" and "get_y" functions, and we have to manually export a function "calculate" with manually created signature.

Instead, extism could generate an idiomatic "binding" based on the protocol, in case of Rust, a trait that looks something like this:

trait MyWorld {
    fn calculate() -> f64;
}

In addition, it could define the external "get_x" and "get_y" functions with correct Rust signature:

fn get_x() -> f64 {
     // call the extern function
}
fn get_y() -> f64 {
     // call the extern function
}

All we would have to do is implement the trait, for example:

struct MyPlugin;
impl MyWorld for MyPlugin {
    fn calculate() -> f64 {
        get_x() + get_y()
    }
}

And done.

Host side

Similarly, when writing a rust host for example, we currently have to specify the name of the function we want to call as string, and pass arguments "dynamically" as a slice of extism::Val. When defining the imported functions, the experience is similar.

Instead, we could generate idiomatic function signatures like so:

trait MyWorldImports {
    fn get_x() -> f64;
    fn get_y() -> f64;
}
struct MyWorldInstance<T> { ... }
impl<T: MyWorldImports> MyWorldInstance<T>{
    fn instantiate(module_bytes: &[u8] import_object: T) -> Result<Self> { ... }
    fn calculate() -> f64 { ... }
}

So instead of relying on manually typing out function names and using dynamic signatures, we get a nice trait where we implement the get_x and get_y method with correct type signatures, and we get an instance from instantiate where we can call the calculate method.

Additional notes

I used a lot of simplification in the code examples, but my basis was this project where I tried to use wit-bindgen, but it didn't work because of some error with the component model version.

More importantly, having idiomatic bindings like this is not only better developer experience to use, but also (and perhaps more importantly), when we change the protocol, we get syntax errors instead of runtime errors in places where we forget to update the code.

And of course, add these idiomatic bindings for every language, not just Rust. For example, in Java/C#, use interface instead of a trait, etc.

What format to use

If you decide that this is something you want for extism, then next question would be what protocol format to use. One option would be to generate the host bindings automatically from the plugin code, or vice versa, but I do not think this is the right solution.

Using .wit format directly as-is might also not be optimal - since I'm not sure if extism supports all features of wit, like resources. Also, if you are using the wit format anyway, then you might as well use wit-bindgen instead.

I think a best solution would be to create a new protocol format designed for extism.

[bhelx]

Obviously this would be really cool and a nice addition to the developer experience. It's something I've thought about doing myself when we first created Extism. I feel like it would be hard to justify coming up with our own IDL though.

One question I'd like to explore is how to support just WIT and use as much of their tooling as possible. My concern, which I have not researched yet, is that by doing that we'd be forcing people into using the component model which currently has it's own binary structure as well as it's own ABI. I've just yet to explore how that would work, but would love some implementation details on that front. So while I support the general idea of providing a layer on top of Extism that gives you a typed interface, we can't accept it at the expense of forcing people to use it.

One other possibility we've discussed is something like gRPC, It's perhaps possible re-use a lot of the tools and code generation around that while still remaining decoupled from it and keeping our simple "bytes-in bytes-out" interface.

[chances]

Regarding your "something like gRPC" comment, I've been considering layering MessagePack on top of Extism.

[bhelx]

That makes a lot of sense. Let us know what you find. Actually I saw @zshipko is working on a crate that handles generic serialization and I think messagepack was supported. Maybe he can chime in.

[zshipko]

Yeah I think Messagepack is a nice target encoding since almost every language already has bindings. I will post a link to the encoding stuff I'm working on once it's a little farther along!

[ericsampson]

There's some similarities to the discussions/proposals here to this EIP
#15