Investigate wasm3 stack overflow during function compilation

[ayazhafiz]

This test currently hits a stack memory limit while compiling a function inside wasm3:

https://github.com/rtfeldman/roc/blob/d4e81ccbd2f2c94725eb21af58cb948b33247898/crates/compiler/test_gen/src/gen_abilities.rs#L920-L940

I think the only way to resolve this, as far as I found, is to compile wasm3 from source ourselves. See wasm3/wasm3-rs#22

[brian-carroll]

This test currently hits a stack memory limit while compiling a function inside wasm3:

I'm not 100% sure this is the right interpretation of the error message you got!

wasm3's error messages are sometimes a bit misleading. I got a very similar error message while working on the dev backend. The real issue was that I hadn't pushed enough values onto the Wasm stack machine but it was reported as an underflow in the "compiling function stack". To me that sounds like an issue inside wasm3 but in fact it was in the app.
So before we go building wasm3 from source I think it's worth doing a bit more investigation to confirm what the problem is.

[brian-carroll]

Also I'm confused about the code comment saying it fails on gen-wasm. I thought you had mentioned before that it was working there. Did that change?

[brian-carroll]

This test passed for me just now on trunk for both gen-wasm and gen-llvm-wasm. I'm on an M1 Mac, which shouldn't matter but maybe does? I can try on x86 WSL Ubuntu tomorrow.

[folkertdev]

the test also passes for me with gen-llvm-wasm on x86-64 linux mint (effectively ubuntu)

[brian-carroll]

Cool. Also works for me on x86-64 WSL Ubuntu with both gen-wasm and gen-llvm-wasm.

[ayazhafiz]

It looks like the error only happens when built in release (cargo test-gen-llvm-wasm --release). In non-release it builds fine. I'm happy to debug this but don't know much about wasm or our infrastructure around it (for dev or llvm); any tips on where to start?

[brian-carroll]

Aha! OK well I thought this issue had gone away but apparently not!

I can look into it, so you don't need to worry about it unless you're interested in learning about it!

My approaches to investigate things like this are

• Load up the test .wasm file in Chrome and see if it gives any errors. We have a webpage for this at crates/compiler/test_gen/src/helpers/debug-wasm-test.html with instructions in the HTML itself. You need to serve that page from localhost.
• Try it out with a different CLI runtime like Wasmer and see if that gives any errors
• Run some analysis tools on it like wasm-validate and wasm-objdump