How does the exterior dialect work?

[jumerckx]

Hey @jackalcooper! I stumbled on this project as I'm working on similar stuff, in Julia. I'm much less far along, though, and this repository is extremely helpful and wonderfully documented. Thanks!

Specifically, I'm also looking into using an out-of-tree/exterior dialect in combination with the C api.
I've built the standalone example dialect and generated a separate C-libary, when I try to use this dialect from within Julia I run into trouble.

# `API` is the default CAPI generated by MLIR (libmlir.so).
# `Standalone` uses the CAPI generated by the standalone dialect example.
using MLIR: API, Standalone

ctx = API.mlirContextCreate() 
registry = API.mlirDialectRegistryCreate()
API.mlirRegisterAllDialects(registry)
API.mlirContextAppendDialectRegistry(ctx, registry)
API.mlirDialectRegistryDestroy(registry)
API.mlirContextLoadAllAvailableDialects(ctx)
handle = Standalone.mlirGetDialectHandle__standalone__() # Note that this function is from Standalone module!
# <--- Everything runs fine till here --->
Standalone.mlirDialectHandleRegisterDialect(handle, ctx) # ERROR: LLVM error: Trying to register different dialects for the same namespace: builtin
API.mlirContextLoadAllAvailableDialects(ctx)

# I also tried using registering the dialect using `API` instead of `Standalone` but this errors as well of course:
#--------------------------------------------------------------------------------------------------------------------#
API.mlirDialectHandleRegisterDialect(handle, ctx)
API.mlirContextLoadAllAvailableDialects(ctx) # ERROR: LLVM error: can't create Attribute 'mlir::StringAttr' because storage uniquer isn't initialized: the dialect was likely not loaded, or the attribute wasn't added with addAttributes<...>() in the Dialect::initialize() method.
#--------------------------------------------------------------------------------------------------------------------#

Did you ever run into similar issues, it seems like both CAPI's try to load a different builtin dialect?
I noticed that in your code, mlirGetDialectHandle__elixir__ is actually part of the CAPI module itself. Does that mean this dialect is included in a monolithic MLIR build?

I actually got around this issue by replacing all calls to API.* methods with Standalone.*. This works, but extensibility is lost. e.g. when a second standalone dialect is introduced AnotherStandalone, I can't just call into this module since this module is not aware of the first Standalone dialect.

I'm curious to hear your thoughts!

Jules

[jackalcooper]

Hi Jules, glad to hear from you.

I noticed that in your code, mlirGetDialectHandle__elixir__ is actually part of the CAPI module itself. Does that mean this dialect is included in a monolithic MLIR build?

yes, it is built and linked with Beaver's native library and I am still figuring out how to properly separate it.

I've built the standalone example dialect and generated a separate C-libary, when I try to use this dialect from within Julia I run into trouble.

is this separate C-libary linking your LLVM libraries? or it is calling add_mlir_aggregate (in my case) so it has its own copy of LLVM/MLIR symbols?

[jumerckx]

is this separate C-libary linking your LLVM libraries? or it is calling add_mlir_aggregate (in my case) so it has its own copy of LLVM/MLIR symbols?

Looking at this code, I guess in my case add_mlir_aggregate is also used when generating Standalone. So in guess your MLIR.CAPI is roughly the same as my (badly named) MLIR.Standalone.

I'm really not yet well-versed in this whole CMake build stuff so I haven't yet found a way forward.

[jackalcooper]

Is it possible for you to load the library produced by add_mlir_public_c_api_library instead of the one produced by add_mlir_aggregate?

[jumerckx]

I have not been able to yet. As I see it, this produces a static archive (.a). And not sure how to change this.

I've written this post on the llvm forum also to discuss this: https://discourse.llvm.org/t/extensible-standalone-capi/71865

[jackalcooper]

maybe add_mlir_public_c_api_library(SHARED ...) could work?

[jumerckx]

I tried that but it didn't seem to change anything. On second thought, I don't think add_mlir_public_c_api was the cause of the .a I noticed. I simply don't see any library produced by add_mlir_public_c_api...
Anyhow, the response on the llvm forum seems to suggest that your approach is how things are generally done so for now I've also just created a single aggregate with everything.