plugin_fn function parameters api

[fliepeltje]

This is more of a discussion/question topic, but I was wondering how difficult/desireable it would be to leverage python function params/annotations in the plugin_fn decorator.

The most straightforward application would be to get rid of the extism.input(...) calls for me, so instead of:

@plugin_fn
def do_it():
  data = extism.input(SomeClass)

You would do something like:

@plugin_fn
def do_it(data: SomeClass):
  ...

Extending that further I could imagine accessing things like config (and other APIs like filesystem operations or host functions):

@plugin_fn
def do_multiple_inputs(input_a: A, input_b: B): 
  ...

@plugin_fn
def do_http_stuff(http: extism.Http, ...): 
  ...

@plugin_fn
def do_with_config(input_a: A, input_b: B, config: extism.Config): 
  ...

@plugin_fn
def do_it(input_a: A, input_b: B, config: extism.Config, http: extism.Http, ...): 
  ...

I'd feel pretty confident implementing something like that in python, but I am not so familiar with pyo3 and how to formulate this in Rust, so I don't know if it is just a massive effort. I think the thing that I'd like to emulate is what axum achieves with extractors: https://docs.rs/axum/latest/axum/index.html#extractors

Ofcourse I understand if there are good reasons for the current API, but I think if you were to say that this is somewhat doable and maybe point me in the right direction, I could explore the API in a fork and we can see how it feels

Looking around I feel I could get quite a way towards making a proof of concept by just messing around in the prelude.py - I just don't know too well how to actually build/test this project. I am able to run ./build.py but I'm unsure what to do next to actually use the compiled library in a test project. From what I can tell there is some import from extism_sys in the sdk that probably (?) is what I want to somehow patch?

[nilslice]

It's a great idea - and totally recognize the desire. We've been working on a whole layer on top of Extism to actually solve this more universally across the PDKs. Check out the "Generating Bindings" section of the README in this repo & let me know what you think.

(I'd link you directly, but I can't seem to do it from my mobile!)

[bhelx]

I generally, in my mental model, see the input args as things that are passed in from the host. These other things are like the std lib for the plugin system. Though I think I could see an argument either way. I'm open to making the python pdk a little bit different if it is more idiomatic to python devs, but i'm not really one. I'm passing to @zshipko to decide as he has made all the decisions in this lib so far.

[zshipko]

I am definitely open to supporting something like this! It's kind of almost there, but we would just need to move some logic into the decorator instead of the function body.

Happy to help or review whatever you come up with!

[bhelx]

@fliepeltje to get some clarity on this part: The most straightforward application would be to get rid of the extism.input(...) calls for me, so instead of:. are you referring to bindings generation? it's a little different than the concrete examples you gave which are effectively system libraries. If you're looking for the former we do have a bindgen system through xtp tools: https://github.com/dylibso/xtp-bindgen

[fliepeltje]

@bhelx (and @nilslice ) I am not referring to bindings generation; I'm really just looking at the API of the extism python library; for me bindings are a whole different beast even though from the docs I see how they would match up with the API that I'm thinking of.

I think when it comes to language-agnostic/universal bindgen (xtp or otherwise) they are difficult to get right in a way that feels intuitive. A different example of a similar take for me comes from the wasm component model: https://component-model.bytecodealliance.org/design/wit.html

I think that while practically it is great to have, any custom DSL will be a pretty big barrier in terms of trying something out and/or learning more. Ideally you'd have both: a language-native API that feels familiar and then when you need it, you can graduate to the DSL, ideally the DSL is made familiar by the API provided by the library.

[fliepeltje]

I've opened a draft to show my general idea

[bhelx]

for me bindings are a whole different beast

i see, just making sure. we agree! that's why we have it as a separate, optional project. we are aware of the component model but it requires a different non-standard bytecode and doesn't work with our philosophy. I just wanted to make sure you're not going down that road as we'd reject it. Will take a look at your draft!

[fliepeltje]

I've updated the PR and resolved the error that I was running into. All arg names are arbitrary, except for input - I think this is gonna be a limitation as long as you can only pass a single arg as input.

You can explore the examples in the PR. For discussion, here's a summary:

Combine multiple extractors in a function

import extism
from dataclasses import dataclass

@dataclass
class Count(extism.Json):
    count: int
    
@extism.plugin_fn
def count_vowels(cfg: extism.Config, input: str) -> Count:
    msg = cfg.get_str("message")
    extism.log(extism.LogLevel.Info, f"Config: {msg}")
    extism.log(extism.LogLevel.Info, f"Input: {input}")

Extract JSON-type input

import extism
from dataclasses import dataclass

@dataclass
class Count(extism.Json):
    count: int

@dataclass
class CountVowelsInput(extism.Json):
    text: str
    
@extism.plugin_fn
def count_vowels_dataclass(input: CountVowelsInput) -> Count:
    extism.log(extism.LogLevel.Info, f"Json Input: {input.text}")

Use the builtin http module

import extism

@extism.plugin_fn
def count_vowels_http(http: extism.Http) -> Count:
    resp = http.request("http://www.example.com")
    extism.log(extism.LogLevel.Info, f"Response: {resp}")