Games are blinking with weird colours in fullscreen mode when using upscaling

[StragaSevera]

Is there an existing issue for this?

• I have searched the existing issues

Are you using any gamescope patches or a forked version of gamescope?

• The issue occurs on upstream gamescope without any modifications

Current Behavior

Whenever I launch a game upscaled with filters (either fsr or nis) and put it in fullscreen mode - either by shortcut or launch option, the screen starts blinking with weird colours and the Steam console log is spammed with [gamescope] [Error] xdg_backend: Compositor released us but we were not acquired. Oh no.
Tried this on two separate games - Graveyard Keeper (native) and Titan Quest (proton) - both have this blinking.
Attaching a photo of the screen, because for some reason a screenshot doesn't capture the artifacts:

Steps To Reproduce

1. Launch Graveyard Keeper with gamescope -w 1920 -h 1080 -F fsr -- %command%.
2. Observe no visual problems and no log spam.
3. Press Super+F to make the game fullscreen.
4. Observe artifacts and log spam.

Hardware information

- Distro: Arch Linux
- CPU: Intel(R) Core(TM) i7-10700F CPU @ 2.90GHz
- GPU: NVIDIA GeForce RTX 4070 Ti SUPER
- Driver Version: NVIDIA 565.77

Software information

- Desktop environment: KDE 6.2.5
- Session type: wayland
- Gamescope version:  gamescope version 3.16.1 (gcc 14.2.1)
- Gamescope launch command(s): `gamescope -w 1920 -h 1080 -F fsr -- %command%`

Which gamescope backends have the issue you are reporting?

• Wayland (default for nested gamescope)
• DRM (default for embedded gamescope, i.e. gamescope-session)
• SDL
• OpenVR

Logging, screenshots, or anything else

No response

[matte-schwartz]

Can you share a full log that starts from launching Gamescope?

[StragaSevera]

Here it is: https://gist.github.com/StragaSevera/c81d3c623b8cf6d41004eafd17008876

[matte-schwartz]

Hmm, I bought the game and I'm not seeing the same type of flickering yet on a laptop I have with a 3070 dGPU but I'll keep trying for a bit. The picture you shared looks very similar to #1593 (comment) though.

[StragaSevera]

Wow, you bought the game just for debugging my problem? o_O
Yes, it looks really similar.

[matte-schwartz]

I do see the issue on my desktop Nvidia rig as well, it started happening after switching between TTY consoles a few times.

I highly suspect it's the same corruption issue as the DRM backend that I linked, since both that issue and your issue started happening at the same time during my testing today. Unfortunately I don't think it's been picked up as an official bug by Nvidia yet so I don't really have any suggestions in the meantime other than stay tuned

[StragaSevera]

I'm using the wayland backend right now, I get errors in console when I try to use DRM or SDL ones. Should I link the logs with those backends?

[matte-schwartz]

SDL backend I already know the issue, you need #1548 which isn't in a gamescope release yet

DRM I'd need to see logs to know what's going on there.

[matte-schwartz]

Flickering seems to come from somewhere in gamescope's composition pipeline, you can trigger the same flickering without FSR just with gamescope -w 1920 -w 1080 --force-composition -- %command% and then toggling the gamescope window to fullscreen.

[matte-schwartz]

not game specific either, but just appears to be NVIDIA-specific.

working: gamescope -w 1920 -h 1080 --composite-debug -- glxgears and Super + F to toggle fullscreen

IMG_2784.mov

flickering: gamescope -w 1920 -h 1080 --composite-debug --force-composition -- glxgears and Super + F to toggle fullscreen

IMG_2785.mov

CCing @cubanismo on this, this issue should be pretty straightforward to replicate on any Wayland DE (I'm using Plasma 6 Wayland) with NVIDIA drivers. I'm on 570.86.16 but past driver versions (and closed modules) are also affected by the issue. I'll try to dig in more to see if I can narrow down what part of our composition process in gamescope might be causing issues here.

nvidia-bug-report.log.gz

[StragaSevera]

DRM I'd need to see logs to know what's going on there.

Here are the logs: https://gist.github.com/StragaSevera/49beb9d54fd0ca74884cb62b2821e769

Yep, the command you listed produces flickering for me too =-)

[cubanismo]

Thank you for the report. For future reference, I've filed NVIDIA internal bug 5089016 to track this issue.

[matte-schwartz]

From what I've been able to gather so far:

The issue happens in gamescope while nested on wayland under both plasma desktop and cosmic desktop, along with gamescope's DRM backend with the same commands as above. If the host compositor is using direct scan-out and nested gamescope is also using direct scan-out, the issue does not occur. If you force composition within gamescope but leave direct scan-out enabled within the host compositor, then you'll start to see the flickering recorded above. Notably, the flickering only starts to occur once gamescope's window is fullscreen on the host compositor.

you can avoid the flickering altogether on nested gamescope by setting an environment variable to disable direct scan-out on the primary compositor (KWIN_DRM_NO_DIRECT_SCANOUT=1 for example). This lets you run both gamescope -w 1920 -h 1080 --composite-debug -- glxgears and gamescope -w 1920 -h 1080 --composite-debug --force-composition -- glxgears without any flickering occurring during the process.

Of course, this doesn't apply to gamescope's DRM backend since force composition is our process for disabling direct scan-out, so once you do that it will immediately start flickering. To make things even more complicated, it is actually possible to sometimes get gamescope composition to work properly in the DRM backend. If you make gamescope-session your first graphical target after plymouth (skipping SDDM for instance) and you start the session with scan-out disabled, no flickering occurs at first. If you enable/disable HDR, or if you switch sessions (ie shutdown steam -> start plasma desktop -> logout and start gamescope-session again), the flickering will start to occur from the moment the session launches.

I still think the commands provided above for nested Wayland desktop are the most consistent way to reproduce the flickering, but maybe some of the additional context will help for narrowing down the issue.

[matte-schwartz]

This is kind of bizarre but the issue also does not occur on nested wayland if you enable HDR on Plasma 6 desktop, but do not enable it in gamescope, ie gamescope --force-composition --composite-debug -- glxgears only with the HDR checkbox toggled in plasma's settings. Then you can fullscreen the window repeatedly without flickering and regardless of composition being used or not.