Lambda support

[mutagene]

I made a draft PR which added support for c++ lambdas reminiscent to how Java/Kotlin does for Single Abstract Method "functional interfaces".

In general, when I'm calling into Java or ObjC and passing a callback or listener, it's with a lambda that I'd like to do this - and since Djinni doesn't support this natively, this results in the use of utility wrapping classes which implement the functional interface that djinni requires. If one is conscientious in naming the functional interface's one method, c++ templates can make it easier to instantiate these wrappers/adapters, but it's still IMO a bit annoying when all you really want is a lambda in the first place.

Anyway, I'm closing that draft PR for now, but would be interested in keeping the lambda discussion alive in the hope that some solution will eventually make its way into Djinni. @eakoli had mentioned in this post work being done on adding full lambda support - which would clearly be great.

[a4z]

Thank you Alexis!
I am super happy you are keeping this topic alive and encapsulate the info to the topic here, since I also think we should have an easy to use solution that avoids creating boilerpalte as much as possible

[eakoli]

I've been working on building lambda support using extern types first, to get the mechanics of everything all working, then transitioning to add support directly in the generator.

Here is where I'm at.

Need:
simplify use of callbacks, which currently involves using interfaces with one method.

Proposal:
introduce a new djinni type function
this new type would be represented as a std::function in C++, block in ObjC and functional interfaces in Java.

Syntax:
formal type declaration (for simple reuse)
<type name> = function( [<args>...] )[: <return type>]

inline parameter syntax
<parameter name> : ( [<args>...] )[-> <return type>]

Syntax examples:

  compare_strings = function( a : string, b: string ): i32;

  complex = interface {
      sort( strings : list<string>, comparator: compare_strings ) : list<string>;
 
      sort_alternate( strings : list<string>, comparator: ( a : string, b: string ) -> i32 ) : list<string>;
  }

Implementation:
It needs to be a value type, thus can be embedded into other records.

For ObjC the implementation is fairly simple, as toCpp just needs to wrap the block into a std::function, and fromCpp just needs to make a block to wrap the std::function.

Java is more complex. It would appear on the java side just as we can do now, generating an interface with one method. It would still need the full proxy object, the toCpp would be able to wrap the proxied object with a lambda and store in the std::function. fromCpp would need to create an implementation of the c++ interface that delegates to the std::function and is returned to java via the normal interface proxy.

This creates two downsides:

• even though we internally treat the function as a value type, on java it will consume entries in the proxy cache. (minor)

• a true java type needs to exist for each function (or at least each unique signature). Using the 'formal type' declarations, that should be simple, as each type is already required to be unique. But for the 'anonymous' type, It creates an issue, as I would think it needs to be a requirement for the interfaces names to be consistent and not impacted by other changes to an interface.

One approach would be to use generics, thus you could make N permutations where N is the max number of args +1, The downside to this approach is that all types need to be boxed.

Another approach would be to create the functional interface names using the same algorithm as the names for Jni functions, this would yield consistent names for the interfaces, but means the interface name would change if the signature changed, potentially causing issues if the interfaces are formally 'implemented' on the java side, as opposed to using the lambda syntax.

In order to defer this hurdle, I've opted to only implement the formal type syntax initially.

Thoughts @a4z @mutagene ?

[a4z]

I would need more time to create a context in my head and unroll the implementation suggestion.

But as a first feedback, I would like to mention, we do not only have ObjC and Java today, but also C# and Python bindings. And soon wasm, the PR exits, see #127 . (Python just works as a concept but not much is missing)

And this is a problem. When adding new types like this, since there is barely one person who knows all the languages good enough to know how to implement them in the various possible bindings.

On the other hand, there are also ideas for a radically re-structure of some things, and when that happens, extendability like here suggested could be incorporated into the design from the begin.
The topic came up and is now handled under a different caption, see the end of #94.

So I am totally open for all ideas, also very much for this one.

But we need to find a way to incorporate that, and not forget the original goal of the project. That is, support and maintain the status quo, so existing projects that use (the original) djinni have a place where maintenance and bugfixing happens.
But we need also to be open for new ideas and possibilities like this one.

So maybe it's time to extend,

• have generator and support lib focus on maintenance of the the status quo, so existing users do not need to fear breaking changes.
• create a new project in the organisation , maybe djinni2, that is open for all the new language bindings, ideas like this, but also enables to clean up a lot of things that make development complex.

I think this would be the way to go, but that would need consensus of several persons.

[eakoli]

For any language that doesn't have the concept of a lambda/block etc they would continue to use the functional_interface, so the status quo would still be maintained.

The 'middle' part with conversion from/to a std::function is where most of the 'magic' happens, which im trying to implement in a way such that initially there are no changes needed to the non C++ side of the generator, it all gets encapsulated on the C++ side as a transparent type conversion, then if a language supports the concept of a lambda/block it can get updated to expose the type in that manner, as opposed to the functional interface.

the intent is that this is entirely an 'opt-in' feature, to prevent the possibility of breakage

[jothepro]

Another approach would be to create the functional interface names using the same algorithm as the names for Jni functions, this would yield consistent names for the interfaces, but means the interface name would change if the signature changed, potentially causing issues if the interfaces are formally 'implemented' on the java side, as opposed to using the lambda syntax.

IMO that's a behavior that would be tolerable if documented properly.

Can you elaborate on how you would apply the JNI name mangling algorithm to the inline lambda declaration? Do you propose to use the parameter name of the lambda as a replacement for the type name of the formal type declaration?

Given your example:

complex = interface {
      sort_alternate( strings : list<string>, comparator: ( a : string, b: string ) -> i32 ) : list<string>;
  }

The only approach that I can come up with is to use the param name "comparator" as type name, maybe in concatenation with the interface and function name to make it unique.

The Java interface would then be sth like:

@FunctionalInterace
interface ComplexSortAlternateComparator {
   int comparator(String a, String b);
}

Is that what you have in mind?

[eakoli]

I thought about using the class/method/parameter name, but then if any of those names changes, it would result in a different type. And then you also have what if there are multiples in the same method (like a parameter and a return value). I think the name needs to be as consistent as possible such that only changing the signature would cause a name change, as that would also always (presumably) need a code change to handle the altered parameters

From Oracle's documentation:

Dynamic linkers resolve entries based on their names. A native method name is concatenated from the following components:

the prefix Java_
a mangled fully-qualified class name
an underscore (_) separator
a mangled method name
for overloaded native methods, two underscores (__) followed by the mangled argument signature

so something like Functor_<mangled signature>

[eakoli]

From a code generation perspective it would also make sense to standardize on single method name (like invoke) so that things can easily be managed by templates. I already have a translator template that can apply to any functional interface type (provided it uses the method name invoke) to map from the generated C++ proxy interface <--> std::function. No generated code needed for the conversion, all via variadic templates