[vm/ffi] Add support for weak symbols @Native(weak: true)

[simolus3]

In the current dart:ffi system, it's possible to deal with different versions of libraries we're binding to by checking for the existence of symbols:

final lib = DynamicLibrary.open('libhelper.so');
if (lib.providesSymbol('my_function_v2')) {
  // Bind to new implementation
} else {
  // Bind to older implementation
}

It looks like no similar functionality exists with the new @Native annotations. While this might not be a big issue when we control the version of the library through native_add_library, that's not always guaranteed and we might be accessing a library from an operating system.

For this, I think the ability to check whether the symbol a @Native annotation points to is accessible would be nice. This could be exposed with nullable functions:

@Native()
external final void Function()? my_function_v2;

But that doesn't really translate to native globals which are already getters. So a better approach may be to:

1. Allow a guarantee that binding to a symbol that doesn't exist only throws when that function/field is first accessed (e.g. @Native(lazy: true) external void my_function_v2();).
2. Add an API similar to Native.addressOf that checks for the existence of a symbol, e.g. bool exists = Native.exists(my_function_v2). This could also work for globals since the call to Native.exists would be lowered in the CFE.

I'm happy to contribute this feature once there's an basic agreement on the API / whether this is something we should support.

[dart-github-bot]

Summary: User requests optional symbol support in dart:ffi's @Native annotations. Currently, checking for native symbol existence isn't possible with @Native, unlike DynamicLibrary.providesSymbol.

[dcharkes]

• Allow a guarantee that binding to a symbol that doesn't exist only throws when that function/field is first accessed (e.g. @Native(lazy: true) external void my_function_v2();).

This might be better controlled by the LinkMode in the CodeAsset. DynamicLoadingBundled / DynamicLoadingSystem is guaranteed to be lazy. If we add support for StaticLinking, DynamicLinkingBundled andDynamicLinkingSystem then these would try to run the linker against the (tree shaken) Dart code. However, so far we've not moved in such direction.

• Add an API similar to Native.addressOf that checks for the existence of a symbol, e.g. bool exists = Native.exists(my_function_v2). This could also work for globals since the call to Native.exists would be lowered in the CFE.

Native.exists SGTM. 👍

cc @mkustermann thoughts?

[dcharkes]

@liamappelbe Would this be useful for the MacOS/iOS APIs as well? Or do you have a different way of guarding against invoking a function that doesn't exist in a version of the OS?

[liamappelbe]

I don't think it will be useful for ObjC, because all the class loading and method invocation goes through the ObjC runtime API (objc_getClass, objc_msgSend etc). For example, objc_getClass returns null if the class doesn't exist.

[simolus3]

I think I need some guidance on the implementation too. I see that resolving symbols ultimately happens through dart::FfiResolve which throws if a lookup error occurs. It looks like we'll probably need a variant of that function that doesn't throw.
Should there be another argument on FfiResolve, Native._ffi_resolver_function and Native._get_ffi_native_resolver to control that behavior?

Alternatively, since dart::ThrowFfiResolveError throws an argument error, we could also introduce a subclass of ArgumentError, e.g. NativeSymbolNotFoundError, that would be thrown when looking up symbols that don't exist. That way, Native.exists(a) could be lowered into:

try {
  Native.addressOf(a);
  return true;
} on NativeSymbolNotFoundError {
  return false;
}

I don't think that's too bad since symbols usually exist and we don't have to introduce a new argument to the resolver (which I suppose would also mean two different CachableIdempotentCall entries?).

[dcharkes]

Alternatively, we could add

static bool FfiSymbolExists(Dart_Handle asset_handle,
                           Dart_Handle symbol_handle) {

DEFINE_NATIVE_ENTRY(Ffi_GetFfiNativeExists, 1, 0) {
  return Pointer::New(reinterpret_cast<intptr_t>(FfiSymbolExists));
}

It would probably be best if FfiSymbolExists throws an argument error if the asset id is not in the mapping or the asset id mapping is wrong (the dylib doesn't exist). This would require some refactoring of FfiResolveInternal to return the errors for symbol lookup separately from dylib opening and asset id errors.

Also, we probably should also add a Native.assetExists, so that you can look up if an asset is available. (The equivalent of doing File(dylibPath).existsSync() that you would use when doing DynamicLibrary.open(dylibPath). This would enable a package having a fallback on a Dart polyfill if the native code is not supplied. cc @mkustermann

If we believe Native.assetsExists(...) and Native.symbolExists(...) would be on the hot path, we might want to cache those two calls with cachable idempotent as well.

cc @mkustermann for thoughts.

[mkustermann]

Conceptually our declarative ffi bindings are modeled after C (basically a C DSL in Dart): Imagine the Dart code wasn't Dart but C code. The idea being that we'll have similar optimizations to what C code would get (e.g. tree shaking, direct calls, etc). The compiler would have the option (we don't do that atm, but we could) to compile the Dart code to compiled code with unresolved symbols and one could statically link in the corresponding C code, making the resulting shared library or executable emit direct calls to the corresponding functions.

So if we made our compiler emit such code, what may happen with those symbols that don't exist is that either we

• get a linker error when we link the two parts together or
• the resulting shared library would have unresolved symbols, which may cause a dlopen() to fail at runtime due to missing symbols

((**) One could add support to tell the dart native compiler the available native symbols when static linking is enabled (since we may know the static libraries we're later going to link the dart code against). That would allow the dart native compiler to tree shake the code for non-existent symbols (i.e. it would evaluate the Native.exists() at dart compile-time). Though this seems very hackish as normally one could compile native code and dart code in parallel and link afterwards, this would require dart code to be compiled after all native libraries)

So it may make sense to tell the compiler that the symbol is special, it may not exist, so it cannot emit direct calls to it.

If the code is checking for a symbol presence, we have to prevent that symbol from being tree shaken (in the case when the native library is compiled & tree shaken). So we should use a declarative way that works with tree shaking and not a string-based lookup approach.

So I may be in favor of

@Native<Void Function()>(symbol: "foo", optional: true) // <-- optional may cause it to throw
external void foo();

@Native<Int8>(symbol: "bar", optional: true) // <-- optional may cause it to throw
external int bar;

if (Native.exists(foo)) { ... } // <-- we only allow referring to declarative symbols and only those that are `optional: true`
if (Native.exists(bar)) { ... } // <-- same

Right now all our resolution is lazy, so lookups are done lazily and errors get thrown if lookup fails, so optional: true wouldn't be necessary with current implementation, but as mentioned above if we did emit direct calls the compiler would need to know that the symbol may not exist and it has to treat it specially (and it may not be practical for the build tool to tell the compiler which symbols are available).

The next thing is that our symbol resolution currently is multi-stage IIRC: It looks in one place and if cannot find it there it falls back to process-wide symbols (@dcharkes may be able to confirm is this is true). We'd probably not want to do this fallback pass for such optional symbols: The developer intend is to say: This symbol should be in a specific library and if it's not in there I do something else - so it shouldn't follow any fallback to process-symbol lookups.

Also, we probably should also add a Native.assetExists, so that you can look up if an asset is available.

If the Dart code works without any of the native assets, that would make all the native symbols optional, which in return would never allow the Dart compiler to emit direct calls to targets (modulo the (**)).

[mkustermann]

In the current dart:ffi system, it's possible to deal with different versions of libraries we're binding to by checking for the existence of symbols:

@simolus3 If you wrote this kind of code in C instead of Dart with FFI, how would the C code look like?

Maybe this is a case where your C code doesn't know what it compiles against, so it would use dlopen(), dlsym(), ... to access those symbols in a dynamic way.

So if that's how the C code would look like, then I would assume you also would use the dynamic FFI api we have (not the declarative ffi api). Though the dynamic API we have - using DynamicLibrary.open() etc doesn't integrate with native assets yet.

Maybe what this is asking for is something like this:

@NativeLib(assetId: ...)
external DynamicLibrary? libfoo;

if (libfoo != null) {
  if (libfoo.providesSymbol('foo')) { ... }
}

That would give developers access to native-asset bundled libraries via the dynamic FFI api. Then a developer would be responsible to ensure the hook is only adding dynamic libraries (and never static ones) and doesn't tree shake it.

[simolus3]

Also, we probably should also add a Native.assetExists, so that you can look up if an asset is available.

I like this idea 👍 It would also allow package authors to include both native-assets based code and older code based on DynamicLibrary lookups, and then use Native.assetExists to check whether the new implementation can be used or whether to fall back to the old one.

If you wrote this kind of code in C instead of Dart with FFI, how would the C code look like?

In my case, I'm talking about these functions for package:sqlite3 (and I need to check whether sqlite3_prepare_v3 is available). SQLite provides a preprocessor definition for the version, so C code could use:

#if SQLITE_VERSION_NUMBER > 3020000
 sqlite3_prepare_v3(...);
#else
 sqlite3_prepare_v2(...);
#endif

Obviously that can't be translated to Dart.

so it would use dlopen(), dlsym(), ... to access those symbols in a dynamic way.

I'm fine with using DynamicLibrary for these optional functions, but:

1. Once the linker is integrated with Dart's tree-shaking system, I need a way to express that I need these functions when they're available (but optional: true would work).
2. There is no way to get a DynamicLibrary for a Dart file using @Native declarations.

[dcharkes]

((**) One could add support to tell the dart native compiler the available native symbols when static linking is enabled (since we may know the static libraries we're later going to link the dart code against). That would allow the dart native compiler to tree shake the code for non-existent symbols (i.e. it would evaluate the Native.exists() at dart compile-time). Though this seems very hackish as normally one could compile native code and dart code in parallel and link afterwards, this would require dart code to be compiled after all native libraries)

This relates to a deeper problem that we have with native API evolution and how to write Dart code that works against different version of a native API. dart-lang/native#1588 dart-lang/native#1590

If we'd know what version of the native API we'd be having at runtime when compiling Dart, we could do something smart. For example use a Dart define and let the AOT tree shaker take care of tree-shaking all calls for the different native API version.

However, if we don't know at compile time what version of the API we will have at runtime, we need checking at runtime. This could either be indirectly through seeing if a symbol exists, or more explicitly via an API call that returns a version. Now, obviously that means we wont be doing static linking!

So we're either (A) static linking and then we could know what version of the native API we have, or (B) we're dynamically loading and we don't know what version of the native API we have.

With native assets, we'd know which assets have (A) static linking and which once have (B) dynamic loading.

and it may not be practical for the build tool to tell the compiler which symbols are available

I don't believe it would be needed to tell which symbols are available. Symbols not being available would only enable us to compile a throw into the code instead of an unresolved symbol

Instead, the Dart compiler needs to know which assets are statically or dynamically linked + a set of defines which should remove the dead code that contains the calls to native code which are not available.

This agrees with the approach if it were C instead of Dart code, you would pass the same define to both builds, or have it be defined by importing a header file, if you have scenario (A). And if you have scenario (B), you're likely using dlsym.

I do agree that trying to support (A) with actual static linking might be impractical. Wiring up the Dart build to get the right defines could be painful.

(@mkustermann we have a bigger issue w.r.t. static linking. We want to tree-shake native assets, the link hooks must run after Dart AOT compilation. If you want to compile Dart code to call external symbols, you must first get the assets list before doing Dart compilation so that you know what assets are statically linked. Which means the link hooks must run before Dart compilation. With us moving asset resolution to embedder callbacks for Flutter, there's no way for the Dart compiler to access which code assets must be statically linked and which ones are dynamically linked. The NativeAssetsApi assumes dynamically loaded. And we can't put whether it's statically linked in the @Native annotation either, because then it would stop working in JIT mode. We'll likely always use dynamic loading for JIT.)

A side question here is how FFIgen configurations should work. If you run FFIgen on your API (with a certain set of defines), it will only output the bindings for a single version of the API. It would be neat if you could run FFIgen multiple times with different sets of defines, and it would automatically spit out the optional: true part.

The next thing is that our symbol resolution currently is multi-stage IIRC: It looks in one place and if cannot find it there it falls back to process-wide symbols (@dcharkes may be able to confirm is this is true). We'd probably not want to do this fallback pass for such optional symbols: The developer intend is to say: This symbol should be in a specific library and if it's not in there I do something else - so it shouldn't follow any fallback to process-symbol lookups.

Yes, we don't want to do process lookup as fallback. @mkustermann we also check GetFfiNativeResolver from Dart_SetFfiNativeResolver, should that be supported?

Maybe what this is asking for is something like this:

@NativeLib(assetId: ...)
external DynamicLibrary? libfoo;

if (libfoo != null) {
if (libfoo.providesSymbol('foo')) { ... }
}

That would make your Dart code imply that you can never use static linking. That's okay. It means you're in scenario (B). Native.assetExists and Native.symbolExists would imply the same thing.

For supporting scenario (B), this solution is simpler than introducing Native.assetExists and Native.symbolExists. 👍

@simolus3 Are you in scenario (A) or (B) ?

If you're in scenario (A), I'd expect your Dart code to be somewhat like

void main() {
  const String sqliteVersion = String.fromEnvironment('SQLITE_VERSION_NUMBER');
  final int = sqliteVersionInt = int.parse(sqliteVersion);

  if (sqliteVersionInt > 3020000) {
    sqlite3_prepare_v3();
  } else {
    sqlite3_prepare_v2();
  }
}

@Native()
external final void Function() sqlite3_prepare_v3;

@Native()
external final void Function() sqlite3_prepare_v2;

And to compile it with flutter build --release --dart-define=SQLITE_VERSION_NUMBER=3090000 for AOT mode and flutter run --debug --dart-define=SQLITE_VERSION_NUMBER=3090000 for JIT mode.

And the hook could then output a dylib in JIT mode (linkingEnabled=false) and a static library in AOT mode.

Passing those Dart defines in manually is not very practical though.

[simolus3]

Are you in scenario (A) or (B) ?

Well, I'm writing a library, so I think I want to be agnostic wrt. to the linking mode and let users make that decision (if at all possible, I'm probably not understanding all of the fine details here). That also means it's very hard to pass the version number along. From the look and feel, I'm definitely leaning towards (A).

At the moment, my logic looks like this (using DynamicLibrary.lookup instead of @Native, but I hope to translate the concepts when porting my library):

void main() {
  if (sqlite3_library_version() > 3020000) {
    sqlite3_prepare_v3();
  } else {
    sqlite3_prepare_v2();
  }
}

@Native<IntPtr Function()>()
external int sqlite3_library_version();

@Native()
external void sqlite3_prepare_v3();

@Native()
external void sqlite3_prepare_v2();

From my understanding, the only issue with that setup is that this may cause linker errors with static linking if we're linking against a libsqlite3.a that doesn't have sqlite3_prepare_v3? If we have an optional: true on @Native() that would silence these errors and turn the call into a runtime error, that would work for me.

[dcharkes]

If you have a libsqlite.a it should come with libsqlite.h, and it will have #define SQLITE_VERSION. https://www.sqlite.org/c3ref/c_source_id.html

Then you are in case (A) static linking, and in C you would use:

#if SQLITE_VERSION_NUMBER > 3020000
 sqlite3_prepare_v3(...);
#else
 sqlite3_prepare_v2(...);
#endif

(Also, dynamic linking would work with libsqlite.so + libsqlite.h. Dynamic linking also needs to know the symbols at compile time. Let's call that case (A2).)

If you use sqlite3_library_version() in C, and your C code looks like

  if (sqlite3_library_version() > 3020000) {
    sqlite3_prepare_v3();
  } else {
    sqlite3_prepare_v2();
  }

then your C code must be using dlsym to look up sqlite3_prepare_v2 and 3. This is dynamic loading, which is case (B).

Case (B) means that your C code can be used with whatever version of libsqlite3.so is present on some target system and you don't bundle it in your app.

Well, I'm writing a library, so I think I want to be agnostic wrt. to the linking mode and let users make that decision (if at all possible, I'm probably not understanding all of the fine details here).

So, I don't believe this is possible. If you'd be writing C and you'd also have to pick whether you use #if or if( on the sqlite version. And this would either enable static linking, or it would enable the C code to work with arbitrary versions of the dylib.

Given that you have written if (sqlite3_library_version() > 3020000) as runtime branching, static linking cannot work, in C or Dart code. So, whether it's DynamicLibrary or Native used in Dart, it will always be dynamic loading. Code written in this way can never be made to use static linking.

If you'd like static linking to work, we need the equivalent of > #if SQLITE_VERSION_NUMBER > 3020000 in Dart. That is using --dart-define=SQLITE_VERSION_NUMBER=3090000 and String.fromEnvironment('SQLITE_VERSION_NUMBER'). I don't see another way to achieve static linking with APIs that depend on defines.

Does this make sense? (Kudos to @mkustermann for "how would the C code look like?" to analyze the situation.)

If you have if (sqlite3_library_version() > 3020000), and you want to use dynamic loading, why do you need a Native.symbolExists? Why not branch on sqlite3_library_version?

[simolus3]

So, I don't believe this is possible.

It appears to be possible with weak symbols (TIL!) in C, and I think it would be nice to have the same thing in Dart.
When linking dynamically it doesn't make a difference, and with static linking it's possible to use linker scripts to replace the symbol with a null pointer if it wasn't found. So I think a @Native(weak: true) is pretty much exactly what I want.

and you want to use dynamic loading, why do you need a Native.symbolExists? Why not branch on sqlite3_library_version?

Fair point, I don't need Native.symbolExists as a runtime condition here.

[dcharkes]

TIL! ❤

and with static linking it's possible to use linker scripts to replace the symbol with a null pointer if it wasn't found

Yes, the Dart equivalent of that would be indeed:

Symbols not being available would only enable us to compile a throw into the code instead of an unresolved symbol.

Edit: Or actually compile a weak symbol into the AOT snapshot. 😄 (Though that would lead to a segfault instead of a throw if accessed at runtime. Depends if we want the speed or safety.)

Once we get to static linking, we should indeed consider @Native(weak: true). Given that for now you cannot report StaticLinking in CodeAssets at all in Dart and Flutter. And there are numerous other problems to solve with static linking (1. we use a custom ELF format for snapshots, even on non-Linux, and 2. once you run the native linker against our snapshots our custom ELF loader doesn't respect all extra sections). So we'll get to that another day.

(We have a bigger issue w.r.t. static linking. We want to tree-shake native assets, the link hooks must run after Dart AOT compilation. If you want to compile Dart code to call external symbols, you must first get the assets list before doing Dart compilation so that you know what assets are statically linked. Which means the link hooks must run before Dart compilation. With us moving asset resolution to embedder callbacks for Flutter, there's no way for the Dart compiler to access which code assets must be statically linked and which ones are dynamically linked. The NativeAssetsApi assumes dynamically loaded. And we can't put whether it's statically linked in the @Native annotation either, because then it would stop working in JIT mode. We'll likely always use dynamic loading for JIT.)

This actually could potentially work if we do the order of 1. build hooks, 2. dart->kernel(AOT mode with tree shaking), 3. link hooks, 4. kernel+assets->aot snapshot.

For your current situation you can rely on @Natives being "lazy" due to DynamicLoadingBundled or DynamicLoadingSystem link modes. And use of sqlite3_library_version() > 3020000.

Is there anything else you need from us at the moment?

I'll convert this issue to tracking adding @Native(weak: true).

[mkustermann]

I don't believe it would be needed to tell which symbols are available. Symbols not being available would only enable us to compile a throw into the code instead of an unresolved symbol

That's a big difference as one makes the app fail to build the other would fail at runtime (only if executed - which it may not be if the code has some guard against it).

Instead, the Dart compiler needs to know which assets are statically or dynamically linked + a set of defines which should remove the dead code that contains the calls to native code which are not available.

This agrees with the approach if it were C instead of Dart code, you would pass the same define to both builds, or have it be defined by importing a header file, if you have scenario (A). And if you have scenario (B), you're likely using dlsym.

There's a difference to C: In C only if/def'ed code is included. In Dart whether a unresolved external symbol is emitted or not would depend on the tree shaking / optimizer being smart enough to prune the code.

Which means the link hooks must run before Dart compilation.

That seems wrong, because the link hooks by-definition are tree shaking and need resource-uses / tree shaking info from dart compilation

(@mkustermann we have a bigger issue w.r.t. static linking. We want to tree-shake native assets, the link hooks must run after Dart AOT compilation. If you want to compile Dart code to call external symbols, you must first get the assets list before doing Dart compilation so that you know what assets are statically linked. Which means the link hooks must run before Dart compilation.

The way it may work:

• packages are asked to build code assets (can emit shared libraries or archives)
• packages that make archives can send it to a link script if they want
• a linker can only emit shared libraries
• all packages that (in build hook) emitted an archive without linker, would be candidates to link together with the dart app

It kind of makes sense because the dart app is compiled by the bundling tool - so any linking that happens with the compiled dart code would happen in the bundling tool, not by a 3rd party link hook.

With us moving asset resolution to embedder callbacks for Flutter, there's no way for the Dart compiler to access which code assets must be statically linked and which ones are dynamically linked. The NativeAssetsApi assumes dynamically loaded. And we can't put whether it's statically linked in the @Native annotation either,)

At runtime only use embedder callbacks to resolve symbols for symbols that aren't statically linked in. The statically linked in assets and calls to those would be resolved at compile-link time of the dart app with shared libraries.

Let's make an example: Flutter builds an app. Some packages emit a shared library. Then Flutter compiles the dart app to AOT. It will tell our AOT compiler: please emit unresolved symbols for any ffi function calls to asset-ids X, Y, Z. The Dart will then emit direct calls with unresolved references. The Dart compiler may also remember in the compiled snapshot data that asset ids X, Y, Z were statically linked in and symbols can be found in the library that has the dart app in it. The flutter build will then take all static archives from build hooks and from dart compilation, link them together and telling the linker to tree shake the app.

At runtime the direct calls will just work. An API such as @NativeLibrary(asset-id: x) DynamicLibrary get x could even be made to work, because the compiled app will know that asset id X is in the same library as the dart app, so it can look up symbols in there (assuming they were also exported - thereby marked as non-tree shakable).

We'll likely always use dynamic loading for JIT

Yes, of course, since dart isn't AOT compiled.

we also check GetFfiNativeResolver from Dart_SetFfiNativeResolver, should that be supported?

Package authors don't have to know about this mechanism and cannot use it either. It's purely for Dart VM embedders to use.

That would make your Dart code imply that you can never use static linking. That's okay. It means you're in scenario (B). Native.assetExists and Native.symbolExists would imply the same thing.

See above, the AOT compiler could theoretically remember asset id X was statically linked in, therefore the DynamicLibrary is whatever the library that has the dart app in it as well - possibly even Process.executable. The embedder would know where the dart app is. Yes, it makes it possible to have N asset-ids <-> 1 shared library - but that's ok.

So, I don't believe this is possible. If you'd be writing C and you'd also have to pick whether you use #if or if( on the sqlite version. And this would either enable static linking, or it would enable the C code to work with arbitrary versions of the dylib.

Exactly. If C code is using a unknown sqlite.so file, it would need to use dlsym() to detect presence of symbols. That would map to our DynamicLibrary API in Dart.

It appears to be possible with weak symbols (TIL!) in C, and I think it would be nice to have the same thing in Dart.

Weak linking isn't something in from the C standard. It's compiler extensions where each compiler/linker may have weird rules around how it works. It can sometimes lead to broken linked binaries which crashes at runtime, it's dangerous if not used correctly. I'm not convinced we should be introducing that in Dart FFI.

But what one can always do is to write a Dart package with a hook. The hook can compile some C code with weak symbols against a static archive and then emit such a code asset.

This is useful in general: We don't have to solve everything in Dart FFI. Users will soon be able to have native code in packages and can write C wrappers around APIs, use if/def in those C wrappers if needed etc.

This actually could potentially work if we do the order of 1. build hooks, 2. dart->kernel(AOT mode with tree shaking), 3. link hooks, 4. kernel+assets->aot snapshot.

The fact that dart2native compilation is split into multiple phases is an implementation detail - we should treat it as a black box iff there's no really strong reasons not to. (ideally the resources.json would be emitted by what symbols are used by the final compilation result - i.e. gen_snapshot - we just happen to compute it based on TFA results because it's easy for now, not because it's best)