steamcompmgr cleanups

[sharkautarch]

No description provided.

[sharkautarch]

also w.r.t. calc_scale_factor(), note that on x86 unix ABI, floats (& float-based composites) are returned in simd 128-bit xmm registers, so this PR technically also makes calc_scale_factor() slightly more efficient (in this case, glm::vec2 fits in a single xmm register)

[sharkautarch]

@Joshua-Ashton
BTW, I found that we don't currently do anything w/ g_bLowLatency (targets atom gamescopeLowLatency)
Should I remove it as dead code, or will it be functional in the future?

[matte-schwartz]

ping me when this is ready for testing, I'll validate w/ the drm backend

[sharkautarch]

@matte-schwartz Yeah it's ready for testing now

[matte-schwartz]

this commit breaks gamescope on wayland desktops with both SDL and Wayland backends

terminate called after throwing an instance of 'std::bad_variant_access'
  what():  std::get: wrong index for variant
[gamescopereaper] [Info]  reaper: Parent of gamescopereaper was killed. Killing children.
(EE) failed to read Wayland events: Broken pipe
fish: Job 2, 'gamescope -- vkcube' terminated by signal SIGABRT (Abort)

https://gist.github.com/matte-schwartz/058fe3b75786157e57d55d0a402ad21f

[sharkautarch]

@matte-schwartz
Just pushed out a commit that I think should fix that issue?

[matte-schwartz]

It does fix it

[matte-schwartz]

@sharkautarch couple issues so far.

1: gamescope -e -f -h 1440 -w 3440 -- steam -steamdeck -steamos3' fails to launch on plasma wayland, gives this short coredump: https://gist.github.com/matte-schwartz/b5a25c8d286a2c59b3d1ef41d7e363c9

[gamescopereaper] [Info]  reaper: Parent of gamescopereaper was killed. Killing children.
(EE) failed to read Wayland events: Broken pipe
fish: Job 1, 'build/src/gamescope -e -f -h 14…' terminated by signal SIGABRT (Abort)

X connection to :1 broken (explicit kill or server shutdown).
src/common/framefunction.cpp (238) : Assertion Failed: CFrameFunctionMgr::~CFrameFunctionMgr: non static FrameFunction[CHTTPClient::BFrameFuncHandleCompletedWorkItems] still registered
src/common/framefunction.cpp (238) : Assertion Failed: CFrameFunctionMgr::~CFrameFunctionMgr: non static FrameFunction[CHTTPClient::BFrameFuncHandleCompletedWorkItems] still registered

works as expected on latest gamescope master

2: gamescope w/ DRM backend fails to present the main window properly. like you can hear UI noise when you use a controller or keyboard, but the screen itself stays black. gamescopectl backend_info shows that the window should be visible while it's not. Only thing that seems to show up is the cursor, and the QR code looking debugger squares to it appears that gamescope is still compositing. Not sure what logging would be useful here, I can check in the morning.

the command I'm testing with from a TTY is:
gamescope -e -f -h 1440 -w 3440 -r 175 -- steam -steamdeck -steamos3 -steampal -gamepadui

[sharkautarch]

@matte-schwartz
try building gamescope w/ UBSAN & ASAN w/ these meson flags:
--buildtype=debugoptimized -Db_sanitize=address,undefined -Dc_args="-fsanitize-recover=all -w" -Dc_link_args="-fsanitize-recover=all -w" -Dcpp_args="-fsanitize-recover=all -w" -Dcpp_link_args="-fsanitize-recover=all -w"

and then when running gamescope, add this ASAN env: env ASAN_OPTIONS=halt_on_error=0:detect_stack_use_after_return=1:strict_string_checks=1:detect_invalid_pointer_pairs=4:dump_instruction_bytes=1 gamescope <parameters> |& tee gamescope.log

[sharkautarch]

@matte-schwartz
I'm not sure whether the commit I just pushed out fixes anything

[sharkautarch]

@matte-schwartz
I was testing running the drm backend w/ this PR, and debugging what was causing issue 2
I think I fixed at least that issue, w/ my latest commit 795aa6a

Thru debugging w/ valgrind, I realized that in handle_wl_surface_destroy in wlserver.cpp, the surf pointer could still appear valid after deleteing it, and the solution was to simply set it to null after deleting it.

After that commit, I was able to run <path/to/gamescope>/gamescope -- steam and see the steam UI stuff.

Weird that this bug isn't being triggered on upstream gamescope, tho often the OS won't immediately reclaim freed memory, so I guess we've just been lucky previously...
nevermind, I think valgrind reported a false positive
I ran gamescope -e -- steam -steamdeck -steamos3 -steampal -gamepadui and still doesnt work... for some reason :(

[sharkautarch]

Figured out what was causing the black screen when running gamescope -e -- steam -steamdeck -steamos3 -steampal -gamepadui
This time it is actually fixed

[matte-schwartz]

Gotten sidetracked today but pulling your latest changes has been going okay so far.