Implemented bicubic downscaling

[ruanformigoni]

This PR implements a bicubic filter to remedy issue #692.

Update 10-mar-2023

Nier Automata (2160p)

Current gamescope result:

Proposal:

Need for Speed ProStreet (2160p)

Current gamescope result:

Proposal:

GTA IV (4320p)

Current gamescope result:

Proposal:

Dark Souls II (4320p)

Current gamescope result:

Proposal:

[vic-bay]

Downsampling from 2160p to 1080p looks okay, but everything lower looks weird.
Here is comparison between 1440p and 1080p with FXAA. 1440p looks a bit aliased and pixelated. In my opinion forced FSR downsampling still looks better.

[EndlesslyFlowering]

you should probably look into Catmull-Rom (Bicubic with b=0 and c=0.5, rather than than b=0.33 and c=0.33) for downscaling rather than using normal bicubic. it's sharper and preserves edges the same as the default.

[rKsanu2MMYvypWePtQWM]

I can confirm that resolutions that aren't double the monitor resolution look weird.

[ruanformigoni]

Hey @EndlesslyFlowering , I've included the Catmull-Rom algorithm, thanks for bringing it to my attention. Thanks for the heads up @rKsanu2MMYvypWePtQWM, I've fixed this in the latest commits.

Here's the ranges of are shown below, the numbers represent the resolution multiplier from the monitor resolution and the render resolution, e.g., render at 2160p and display in 1080p == 2160/1080 == 2.

• Range of [1,2) uses FSR Bilinear
• Range of [2,3) uses Catmull-Rom with a mean of 4 samples
• Range of [3,4) uses Catmull-Rom with a mean of 9 samples
• Range of (4,inf) uses Catmull-Rom with a mean of 16 samples

You can toggle the downscaling effect with super + K

[DisplayTalk]

Definitely making really great progress. I think to cap it all off you should add the ability for user to adjust the bicubic parameters of B and C. Catmull-rom has ringing, which increases perceived sharpness but I want to be able to adjust it.

Examples of different bicubic setups with the 2 parameters:

B-Spline(blur only): B1 C0

Hermite(blocking only): B0 C0

Mitchell(balanced minor amounts of blocking, aliasing, and ringing): B0.33 C0.33

Catmull-Rom(balance of ringing and blocking): B0 C0.5

Explanation of different scalers:
https://artoriuz.github.io/blog/mpv_upscaling.html
https://en.wikipedia.org/wiki/Mitchell%E2%80%93Netravali_filters

Image showing the effects of different bicubic parameters on scaled quality:
https://artoriuz.github.io/blog/images/mpv_upscaling/mitchell_survey.png

Also, if it's not too far out of scope, these scalers(especially the ringy ones like catmull-rom or lanzcos) work better when you convert gamma to sigmoid light before scaling, and back to the native gamma afterwards.

Pic showing that with upscaling:
https://artoriuz.github.io/blog/images/mpv_upscaling/kanao/kanao_comparison.png

This applies also to the downscaling(your PR can be used as both an upscaler or downscaler, if it still works like the first iteration)

[ruanformigoni]

@Displaysguy thanks a ton for the explanation and resources! I'll study how to implement this properly and look into the possible method for argument passing you mentioned here.

[DisplayTalk]

Another tidbit, this one is specifically on sigmoidization:

"You may decrease halos and increase perceptual sharpness by increasing the sigmoidal contrast (up to 11.5, say). Higher contrasts are especially recommended with greyscale images (even “false RGB greyscale” that have three proportional color channels). The downside of sigmoidization is that it sometimes produces “color bleed” artefacts that look a bit like cheap flexographic (”gummidruck”) printing or chromatic aberration. In addition, sigmoidization’s “fattening” of extreme light and dark values may not work for your image content. If such artefacts are obvious, push the contrast value down from 7.5 (to 5, for example, or even lower). Setting the contrast to 0 is equivalent to enlarging through linear RGB. (Robidoux, 2012)"

From this(and the MPV scaling article) we know that sigmoidization can lower both light and dark overshoot(overshoot=ringing) with ringy scalers on both upscaling and downscaling. It's always superior to linearization because linearization only accounts for light overshoot and not dark overshoot.

However, it is not without downside. It has negative effects on the image and in certain situations(even where using it makes sense, which is only when using ringy scaling configurations) might not even be a net benefit. Because of this I wouldn't make adding sigmoidization a priority, and if you do add it make sure it is user-adustable and can be disabled(and is not enabled by default).

https://guide.encode.moe/encoding/resampling.html (ctrl-f sigmoid to find the quote)

[specfreq]

I've had a chance to test this out a bit. Here's some test screenshots.

[ruanformigoni]

Hi @specfreq , could you also include a screenshot with the downsampling algorithm disabled?

[specfreq]

@ruanformigoni Here's some new test screenshots.
One thing I'm doing is changing the desktop resolution to match my target scaled resolution (1024x768 in this case) and forcing on vsync through the OS.

Launch options: gamescope -D -f -W 1024 -H 768 -w 4096 -h 3072 -- %command%

Let me know if I don't have it set up correctly.

[ruanformigoni]

Thanks for the followup @specfreq. I did the same as you (change the desktop resolution and downscale) and got way better results, here is a comparison. Did you set the in-game render resolution to 4096x3072 in the resolution settings?

[specfreq]

The resolution is set to 4096x3072

Here's my git pull:

[ruanformigoni]

Are you able to toggle it with super + k?

[specfreq]

I am not able to toggle it with super + k. I'm pretty new to using Git, should I just delete it and reinstall?

[ruanformigoni]

That should be fine, make sure that lines 176-179 of the file src/sdlwindow.cpp are these:

case KEY_K:
	g_wantedUpscaleFilter = (g_wantedUpscaleFilter == GamescopeUpscaleFilter::BICUBIC) ?
		GamescopeUpscaleFilter::LINEAR : GamescopeUpscaleFilter::BICUBIC;
	break;

In your launch command gamescope -D -f -W 1024 -H 768 -w 4096 -h 3072 -- %command%, do could you point it directly to where it is compiled? E.g.: ~/Repositories/gamescope/build/gamescope -D -f -W 1024 -H 768 -w 4096 -h 3072 -- %command%.

Edit: You can also try with other keys, maybe change KEY_K to KEY_P , recompile, and try with super + p. By your screenshots the bicubic filter is not enabled.

[specfreq]

I just rebuilt it and it is working now.

[specfreq]

Here's a test video:
https://youtu.be/mbMsGK45p-U

And high quality:
https://youtu.be/6RVYbmluInk

I'd love to be able to use this with the integer scaling.

[Filoppi]

I'm not too sure about it but, this branch seems unreachable?
You pass in f (x) as a fract of a pixel coordinates, thus it will never be beyond 0-1?

The original implementation, which I reimplemented here
https://github.com/dolphin-emu/dolphin/pull/11999/files#diff-368d3b7ce4f1890976b9f3b3d7f59df06cc906721d2dd882552a8503b3a40d7bR240
actually passed in fract*m (or n) here, effectively using all branches as it spanned from -1 to 2.
Thought to be honest I haven't 100% understood what the algorithm does so maybe my implementation is all wrong.

[ruanformigoni]

Thanks for spotting this, I've updated the code and now it calculates the weights as described here.

[rKsanu2MMYvypWePtQWM]

any news? would love to see this finally merged...

[Filoppi]

I highly suggest to take a look at this more advanced downsampling method:
dolphin-emu/dolphin@ca93a51
It's heavier, but it looks damn fantastic when scaling from a relatively higher resolution. On older games, this would be perfect and very temporally stable.

[misyltoad]

I don't understand why you are still running the whole RCAS pass, surely we don't want sharpening after doing downscaling?

What's the rationale behind doing FSR for bilinear at certain ranges? Does it look better? I have my doubts there.

I think it also makes sense to have a g_wantedDownscaleFilter and g_downscaleFilter. Shouldn't affect upscaling at all. I would be happy to default downscaling method.

Then do like:

				bool needsUpScaling = frameInfo.layers[0].scale.x < 0.999f && frameInfo.layers[0].scale.y < 0.999f;
                bool needsDownScaling = frameInfo.layers[0].scale.x > 1.001f && frameInfo.layers[0].scale.y > 1.001f;
                frameInfo.useBICUBICLayer0 = g_downscaleFilter == GamescopeDownscaleFilter::BICUBIC && needsDownScaling;
				frameInfo.useFSRLayer0 = g_upscaleFilter == GamescopeUpscaleFilter::FSR && needsUpScaling;
				frameInfo.useNISLayer0 = g_upscaleFilter == GamescopeUpscaleFilter::NIS && needsUpScaling;

[LethalManBoob]

Progress update?

[Comptr-user]

Is there a way to get this to work?
I tried following the steps for building, but it refused to build. Using FSR for downsampling did build and run, however it was unusable.

[TacoCake]

Hey, is this PR dead?

[ruanformigoni]

Hello everyone, sorry for not going back to this sooner. I have managed to fix the remaining bugs on the bicubic filter.

Bugs Fixed

1. The B and C parameters now work properly.
2. Image artifacts are no longer present
3. Upscaling now works correctly

Recap on what this PR proposes

This PR introduces a bicubic filter for downscaling, as shown in the pictures
above, it produces images with higher quality than the current approach.

For the filter option, a novel argument was introduce bicubic:

./build/src/gamescope -f --filter=bicubic -w 2560 -h 1080 -- vkcube --width 5120 --height 2160

Also, you can configure the B and C parameters for the Mitchell–Netravali's
filter, e.g.

./build/src/gamescope -f --filter=bicubic:1.0,0.0 -w 2560 -h 1080 -- vkcube --width 5120 --height 2160

You can toggle the bicubic filter with Super + K, if that doesn't work, try grabbing the screen first with Super + G

Comparison with Iterative Bicubic

The current implementation is based on this one, it is more efficient computationally than other approaches I have tested. Here are some image quality comparisons from the current efficient approach with a heavier one, this is a upscaling test from 960x540 to 1920x1080.

Other

I'll take some time tomorrow to rebase the changes to the newest commit. You can compile the current version with:

git clone https://github.com/ruanformigoni/gamescope.git
cd gamescope
git checkout issue-692-bicubic
git submodule update --init
meson setup build
ninja -C build
build/gamescope -- <game>

Update

Rebase complete, compiling instructions updated. I created a dockerfile to simplify the build process.

To compile you just need to have docker installed:

# Create the gamescope folder
mkdir gamescope && cd gamescope
# Download the dockerfile
wget -O Dockerfile https://gist.githubusercontent.com/ruanformigoni/9521d5a2ff06be07d7000718751fae1d/raw/46c4ebe431999bd94c483b84c9ca4501051b7a82/Dockerfile
# Compile
docker build . -t gamescope
# Copy the compiled binary to outside the container
docker run -it --rm -v "$(pwd)":/workdir gamescope cp /gamescope/build/src/gamescope /workdir/gamescope
docker run -it --rm -v "$(pwd)":/workdir gamescope cp /gamescope/build/src/gamescopereaper /workdir/gamescopereaper

[rKsanu2MMYvypWePtQWM]

Thank you so much.

[ruanformigoni]

I have rebased the changes on another branch, unfortunately I am getting a black screen now, probably missed something that was introduced before the rebase. @misyltoad could you take a look at the changes and let me know what you think? Thanks!

[LethalManBoob]

great to see this getting implemented, 4k linux gaming!!

[dustContributor]

Thank you Ruan for continuing to work on this!

[ruanformigoni]

Ok, I managed to fix the black screen issue @misyltoad, and also rebased the project to have no conflicts with master. I have modified how the arguments work to fit more what gamescope is doing right now with the --filter argument; so the usage now is --filter=bicubic or -F bicubic, with arguments it looks like --filter=bicubic:0.3,0.3 or -F bicubic:0.3,0.3. My previous comment was updated with the full command line examples.

[matte-schwartz]

@ruanformigoni nice work on this!

something I've found while testing:

• I'm getting quite a few [gamescope] [Error] xdg_backend: Compositor released us but we were not acquired. Oh no. logs when using gamescope -F bicubic -W 2560 -H 1080 -w 5120 -h 2160 -- vkcube on Plasma 6 Wayland with the gamescope Wayland backend. this does not happen with the upscaling filters in gamescope, and it also does not seem to matter if Gamescope WSI is in use or not. I'm not seeing any obvious causes with gamescopectl log_xdg_backend debug but I haven't looked too far into it yet.

I will try to test some with the DRM backend sometime later today.

[ruanformigoni]

Hi @matte-schwartz , thanks! I do not think this is related to the filter, since I also have this issue with the current implementation of FSR:

[matte-schwartz]

Interesting, was not seeing the same on my rig with FSR although I was not testing in-game quite yet... just with vkcube. Could have just been a desktop compositor issue or something on my end I suppose. In general it's not a huge deal when that error appears but figured I'd make a note of it anyway.

[gnusenpai]

I've found a strange bug. Enabling the bicubic filter destroys the gamma in some games, lowering contrast and making the colors washed out. It seems to affect all the Unreal Engine games I've tried.

EDIT: It appears that this is related to 10-bit formats. Disabling Gamescope's WSI layer restores the gamma, but then you don't have 10-bit color of course.

[arch1t3cht]

Image showing the effects of different bicubic parameters on scaled quality:
https://artoriuz.github.io/blog/images/mpv_upscaling/mitchell_survey.png

(The image link is dead but it's still available in the wayback machine, or e.g. at https://upload.wikimedia.org/wikipedia/commons/thumb/6/6f/Mitchell-Netravali_artifacts.svg/1280px-Mitchell-Netravali_artifacts.svg.png )

Note that this graph (and the survey it's based on) is for upscaling. When downscaling, the parameters can have very different effects (depending on how the kernel is scaled with the downscaling ratio).

[Artoriuz]

Image showing the effects of different bicubic parameters on scaled quality: https://artoriuz.github.io/blog/images/mpv_upscaling/mitchell_survey.png

I realise I'm replying to a very old message, but I just wanted to point out that the information previously contained in the linked page has been somewhat moved to this other page. The content itself is different, but it covers the same topics.

[LethalManBoob]

lets get this merged asap pls, linux gamers deserve hq super sampling :<

[MCPO-Spartan-117]

The help output needs to be updated, there isn't a Wayland keybind for this and Nvidia seems to have issues with this for whatever reason as @gnusenpai said.

[Sp3ci3s8472]

I build the project and installed the binaries to test it on my 7900XTX, sadly it does not seem to run. The version I had before from my package manager (pacman) does seem to work (gamescope-3.15.14-1-x86_64). Not sure what additional details you might need but I still have X11 instead of Wayland.
The command I used is:
gamescope -h 2160 -w 5160 -H 1440 -W 3440 -r 120 -- %command%

When I run it on my terminal without any extra parameters:
[gamescope] [Info] console: gamescope version 6649fd8 (gcc 14.2.1) No CAP_SYS_NICE, falling back to regular-priority compute and threads. Performance will be affected. [gamescope] [Info] vulkan: selecting physical device 'AMD Radeon RX 7900 XTX (RADV NAVI31)': queue family 1 (general queue family 0) [gamescope] [Info] vulkan: physical device supports DRM format modifiers [gamescope] [Info] wlserver: [backend/headless/backend.c:67] Creating headless backend [gamescope] [Info] vulkan: supported DRM formats for sampling usage: [gamescope] [Info] vulkan: Creating Gamescope nested swapchain with format 44 and colorspace 0 gamescope: types/wlr_linux_dmabuf_v1.c:532: feedback_compile: Assertion table_len > 0' failed.
Aborted (core dumped)
`

[matte-schwartz]

@Sp3ci3s8472 that's unrelated, you need #1548

[Sp3ci3s8472]

@Sp3ci3s8472 that's unrelated, you need #1548

Thanks that indeed solved it!