Support X11 surfaces when compiled to Wasm

[KnorpelSenf]

Problem Description

I am compiling wgpu to Wasm for use in Deno. I have created X11 bindings for Deno so that I can open a native X11 window via FFI from the JS context. However, I now cannot render to this window with wgpu because of a panic caused by

wgpu/wgpu/src/backend/webgpu.rs

Line 1214 in 9993f5c

_ => panic!("expected valid handle for canvas"),

Desired Solution

I would like to render to an X11 window even though wgpu is compiled to Wasm.

Alternative Solutions

Implement support for OffscreenCanvas in Deno via denoland/deno#5701

[KnorpelSenf]

@Wumpf does “help required” mean that you need outside contributors to tackle this?

If yes, can you outline roughly what is required here? I'm not particularly familiar with wgpu's code base yet, but I'm interested in understanding how complex of a change this is. There's a chance that I can look into this if that's feasible before becoming an expert.

[Wumpf]

does “help required” mean that you need outside contributors to tackle this?

yep, what it says in the tooltip: it needs help from the community to make this happen :)
This is because it's too far outside of the current focus and it's extremely unlikely that any of the current maintainers will look into this.

This sounds like fairly complex change as the handling of surfaces is rather complex and riddled with strange exceptions and per-platform functions. On the bright side, it is riddled with strange exceptions & platform specific things! So an oddball like wasm supported x11 surfaces could fit right in there. So all in all I don't think you'll have to become an expert to take a stab at it (:

I have no idea what passing down x11 surface handles would look like through Deno, but once you have that in some form, it's best to check the other platform specific paths in the wgpu to wgpu-core binding here. From there you can enter wgpu-core which in this case relatively thinly forwards to wgpu-hal, the actual platform abstraction layer. Here you'll have to handle either or both OpenGL & Vulkan. I highly recommend focusing on just Vulkan for starters and leave OpenGL unsupported.

[KnorpelSenf]

does “help required” mean that you need outside contributors to tackle this?

yep, what it says in the tooltip: it needs help from the community to make this happen :) This is because it's too far outside of the current focus and it's extremely unlikely that any of the current maintainers will look into this.

I see, it seems like I just unlocked a new side quest!

This sounds like fairly complex change as the handling of surfaces is rather complex and riddled with strange exceptions and per-platform functions. On the bright side, it is riddled with strange exceptions & platform specific things! So an oddball like wasm supported x11 surfaces could fit right in there. So all in all I don't think you'll have to become an expert to take a stab at it (:

I'm really just playing around and having fun while making a mess so this sounds like a nice endeavour. Great to hear!

I have no idea what passing down x11 surface handles would look like through Deno

I can already run a Wasm build of wgpu inside Deno and construct an X11 handle and pass it to wgpu. It's pretty straightforward in case you're curious: https://github.com/KnorpelSenf/blitz/blob/b7ed40faa7eb6eda37b108c12d96e5bfb7a92a8d/packages/blitz-js/test.ts

(Disclaimer: very ugly and experimental stuff, I'm trying to fail as fast as possible, please don't judge me by this code)

but once you have that in some form, it's best to check the other platform specific paths in the wgpu to wgpu-core binding here. From there you can enter wgpu-core which in this case relatively thinly forwards to wgpu-hal, the actual platform abstraction layer. Here you'll have to handle either or both OpenGL & Vulkan. I highly recommend focusing on just Vulkan for starters and leave OpenGL unsupported.

Thanks for the pointers! Leaving out OpenGL sounds right, Deno supports WebGPU only and not WebGL so my naïve understanding is that Vulkan APIs should be much easier to support?

[Wumpf]

Deno supports WebGPU only and not WebGL so my naïve understanding is that Vulkan APIs should be much easier to support?

Vulkan should be easier I believe but for different reasons. wgpu-core implements WebGPU (and a whole bunch of extensions) through all of Metal, Vulkan, Dx12 and OpenGL. But OpenGL is more of a fallback with (depending on setup) various limitations and overall the WebGPU API is speced with implementations being based on Metal/Vulkan/Dx12 in mind. For the surface handling that shouldn't be all that relevant for you, but it's overall a bit of a rough place with many awkward workarounds.

Btw. in case you hit a dead end here after all: Something you should also be able to do in your application is to render to texture with wgpu (i.e. never create a surface) and then copy that texture to your x11 surface. That last part can be tricky though because for that you'll have to get the underlying Vulkan/GL texture handle out deal with that manually. And I don't think that's exposed in Deno (it's not part of the WebGPU spec of course and I doubt Deno exposes any of these very unsafe exit points)

[KnorpelSenf]

Fascinating!

The only thing Deno exposes beyond the WebGPU spec is https://docs.deno.com/api/deno/~/Deno.UnsafeWindowSurface.

I guess for a small prototype it could be interesting to render to a texture and then pass the buffer to X11 via FFI. That might be very simple to do actually, but it's obviously a terrible idea for any real project because every rendered frame would be passed from VRAM to RAM and back to VRAM.