Unity Editor fails to enable OpenGL when used with VirtualGL

[dcommander]

The Unity Editor attempts to enable OpenGL but fails to do so when used with VirtualGL. Examining the Editor log with VirtualGL tracing enabled, it appears that the editor requests a set of GLX FB configs with GLX_X_RENDERABLE=1, GLX_DRAWABLE_TYPE=GLX_WINDOW_BIT, GLX_RENDER_TYPE=GLX_RGBA_BIT, GLX_X_VISUAL_TYPE=GLX_TRUE_COLOR, GLX_RED_SIZE=4, GLX_GREEN_SIZE=4, GLX_BLUE_SIZE=4, GLX_ALPHA_SIZE=GLX_DONT_CARE, GLX_SAMPLES=GLX_DONT_CARE, and GLX_SAMPLE_BUFFERS=GLX_DONT_CARE. VirtualGL returns a set of GLX FB configs matching those criteria. Unity then iterates through the GLX FB configs, examining the value of GLX_SAMPLE_BUFFERS for each. The set of GLX FB configs includes some that support multisampling and some that don't, so Unity should be able to find what it wants. However, it apparently doesn't, because it then reports "No valid fbconfig found". After a few rounds of this, it tries to enable Vulkan, which seems to interact poorly with VirtualGL (probably because nVidia's Vulkan implementation has some hard-coded dependencies on their proprietary X module and extensions-- long story.)

The issue above was observed with Unity Editor 2020.3 LTS. I also tried a newer version, which appears to require EGL/X11, but I wasn't able to make that newer version work with the new EGL/X11 front end in the VirtualGL 3.1 evolving build.

[dcommander]

Interestingly, Unity Hub appears to use ANGLE, as Chrome does. Referring to #228 (comment) and #222, I get the same glCreateWindowSurface failed with error EGL_BAD_CONFIG error when I attempt to launch the hub with the EGL back end, and the error goes away (and the VGL logo is displayed in response to setting VGL_LOGO=1) if I pass --use-gl=egl when launching the hub. However, the Editor still falls back to using Vulkan.

[dcommander]

In my testing, Unity Hub 3.7.0 works with VirtualGL if you pass --use-gl=egl to it. Then the 2021.X and later editors seem to fully work after that. The 2020.X editor does not work. (The scene window is blank.) Documenting the workaround and closing this issue unless someone wants to pay me to investigate it further.

Referring to #222 and #229, the issues with Unity are similar to the issues with Chrome v85-111. If you don't pass --use-gl=egl to it, then it uses ANGLE, which I haven't yet figured out how to support with VirtualGL. I can make Unity Hub work with the Chrome hacks documented in #229 (comment), along with --use-angle=gl-egl, but the Editor doesn't appear to use VirtualGL when launched with that configuration.

[dcommander]

The scene window in the 2020.X editor is also blank when it is run on the local display without VirtualGL, so that doesn't appear to be a VirtualGL issue.