Using shared WebAssembly.Memory (SharedArrayBuffer) with non-multithreaded builds?

[donalffons]

Hi,

I get the following runtime error when trying to use a shared WebAssembly.Memory with a non-mulithreaded Emscripten build:

RuntimeError: Aborted(LinkError: WebAssembly.instantiate(): mismatch in shared state of memory, declared = 0, imported = 1).

My use case is this:
I have some C++ code that performs a long-running operation. While this operation runs, the code writes a progress value (from 1 - 100%) to a fixed address in memory ("progressPointer").
I would like to do the following:

1. On the main thread, create a worker
2. In the worker, create a shared WebAssembly.Memory, instantiate my Emscripten-compiled WASM (built using -sIMPORTED_MEMORY=1) and pass the shared memory as wasmMemory
3. In the worker, obtain the "progressPointer" (which I can use as an index to module.HEAP8[progressPointer] to access the progress value) and send both module.HEAP8 and progressPointer back to the main thread
4. In the worker, start the long running process
5. On the main thread, periodically check the value at HEAP8[progressPointer] and present that to the user

I have tried some other things, like passing -sSHARED_MEMORY=1 at link time. From the error messages I get, it seems like it is impossible to use a shared WebAssembly.Memory with a non-multithreaded build. Is that correct or is there a workaround for my issue?

[kripken]

Linking with -sSHARED_MEMORY should support this use case, and it works when I try it on hello world, by adding this right before the check of Module['wasmMemory']:

    Module['wasmMemory'] = new WebAssembly.Memory({
      'initial': INITIAL_MEMORY / 65536,
      'maximum': INITIAL_MEMORY / 65536,
      'shared': true
    });

Perhaps check the wasm file, and see that it declares the memory properly as shared? (Is there a chance you are seeing an old wasm file in the cache?) You should see this:

$ wasm-dis b.out.wasm | grep memory
 (import "env" "memory" (memory $mimport$0 (shared 256 256)))

[kripken]

Does that mean that I have to set -sSHARED_MEMORY=1 during compile time, too?

Yes, object files must be built with atomics support for this. The linker just merges them, basically, it can't add in atomics support at that stage (because atomics can require nontrivial changes of various kinds, I believe).

is there a way that I can pass either WebAssembly.Memory({shared: true}) or WebAssembly.Memory({shared: false}) to Emscripten from JS?

I don't think so. Wasm files have a hardcoded indicator of whether the wasm is shared or not (even if the memory is imported). And also atomics support would either be present or not. I think you'd need two separate builds here.

[sbc100]

Thanks for looking into this, Alon!

When I set -sSHARED_MEMORY=1, I get the following error during link time:

wasm-ld: error: --shared-memory is disallowed by /opencascade.js/build/bindings/STEPCAFControl/STEPCAFControl_GDTProperty.hxx/STEPCAFControl_GDTProperty.cpp.o because it was not compiled with 'atomics' or 'bulk-memory' features.
emcc: error: '/emsdk/upstream/bin/wasm-ld @/tmp/emscripten_s_20sihm.rsp.utf-8' failed (returned 1)

Full list of compile flags: -DIGNORE_NO_ATOMICS=1, -DOCCT_NO_PLUGINS, -frtti, -DHAVE_RAPIDJSON, -Os Full list of link flags: -O3, -sEXPORT_ES6=1, -sUSE_ES6_IMPORT_META=0, -sEXPORTED_RUNTIME_METHODS=['FS'], -sINITIAL_MEMORY=100MB, -sMAXIMUM_MEMORY=4GB, -sALLOW_MEMORY_GROWTH=1, -sUSE_FREETYPE=1, -sLLD_REPORT_UNDEFINED, --no-entry, -sIMPORTED_MEMORY=1, -sSHARED_MEMORY=1

EDIT: Does that mean that I have to set -sSHARED_MEMORY=1 during compile time, too? My build takes a while, so I have to wait a few hours before I can try the change.

Whats happening is the linker is telling that one of your objects contains atomics code that code lowered to non-atomics (because it wasn't built with -pthread). In general its not safe to link such object into multit-threaded programs. If you think you know better then you can override that by passing -Wl,--no-check-features to the linker.

Alternatively you can compile all your code with either -pthread (or -sSHARED_MEMORY or -sUSE_PTHREADS or -matomics -mbulk-memory.. they all do enough to get the compile not to lower-away the atomics).

[kripken]

(If you're sure you don't actually need atomics, though, you might try to hack around this by building without shared memory, then modifying the wasm file manually.)

[donalffons]

I think you'd need two separate builds here [...]
If you're sure you don't actually need atomics, though, you might try to hack around this by building without shared memory, then modifying the wasm file manually.

Alright, that's good to know, I'll keep that in mind. I don't think I need atomics for my use-case, but I'm also not aware of all the implications of such a hack. I want to distribute a multi-threaded version of my project at some point anyways, so I might just restrict the progress indicator functionality to that version.

If you think you know better then you can override that by passing -Wl,--no-check-features to the linker.

I thought I might know it better, tried that flag and realized that I really don't 🙂. The build works fine with that flag, but during WASM startup, I get:

ReferenceError: _emscripten_get_device_pixel_ratio is not defined

That seems weirdly unrelated - maybe that's a sign, that I'm a bit too far off the safe track?!

Thanks a lot for your help!

[sbc100]

That looks unrelated yes.