Unified renderer

[veeenu]

This is a MASSIVE refactor that completely changes the underlying rendering engine, which is now a unified DirectX12 renderer that is then composited over games' windows via DirectComposition, with barely any public API change.

It seamlessly works with Dark Souls III and Elden Ring already, and the integration test cases.

Being such a breaking change, I will shamelessly ping all people that may be affected as I don't have a lot of control over corner cases but really don't want to merge this in before I'm sure I'm not disrupting anything. I would be super grateful if you'd be up for checking this out. I expect some of you may have moved on to other projects, in that case I hope you won't be annoyed by the ping!

Explanation

This PR implements a DirectX 12 renderer in hudhook::renderer::dx12 which renders over an off-screen surface, and then uses DirectComposition to composite the off-screen surface on top of the HWND, completely sidestepping the games' native rendering engine which will run on its own.

Present hooks now only serve the purpose of actually triggering a render and a composition and the device objects aren't touched at all. In fact, most of the hooks' rendering logic except building the actual hooked function implementations is now shared and identical across all hooks.

As a result, hooks are now tiny compared to what they were before and their maintenance is going to be very easy. Similarly, with a single renderer there is much less code to maintain.

Additionally, this will make it extremely simple to add new renderers

Affected issues

As a bonus, this PR should fix the following issues, or at the very least give us a cleaner path to their resolution.

If you are getting pinged, it would mean the world if you could give this branch a spin and report back if it fixed the associated issue.

• Game input not registering correctly #79
  @Godnoken @camas since the renderer does not lock DirectX9 specific resources, it could alleviate/solve the issue. (need to investigate)
• Revamp input handling #94
  @Godnoken I have found a different way of blocking input that doesn't involve rerouting many input functions: just use a transparent overlay window. It's not implemented but I think we could explore that avenue and it could work for all hooks.
• [DX9] Some games do not reset at window resizing #103
  @Godnoken this branch uses WM_SIZE for resize events which should sidestep everything that causes your issue altogether. (might close the issue)
• Allow DirectX hooking using existing device pointer #107
  @earthnuker @SkyLeite this greatly reduces the issue surface; now that the hook code is drastically simpler, we could try a number of solutions, provided this one somehow doesn't solve the issue already. (need to investigate)
• Closes Build wgpu renderer #111.
  We won't need a renderer with multiple backends after all!
• DX9 ImGui unresponsive #137
  @vars1ty not sure about the specific game, but maybe things may have changed enough that this now works correctly. (need to investigate)
• Closes Incompatibility Issue with Sea of Thieves (DX11) #142
  @Thoxy67 this branch should most definitely work for you now.

Thank you all for your patience and sorry again for the ping! 😅

[camas]

Unfortunately #79 still an issue. Tested with 7791c49

Fails soon after injection when it can't lock the render engine, with a bunch of WndProc lock fails before.

19:00:12 [DEBUG] (1) hudhook::hooks: WNDCLASSEXW { cbSize: 80, style: WNDCLASS_STYLES(3), cbClsExtra: 0, cbWndExtra: 0, hInstance: HINSTANCE(140696328601600), hIcon: HICON(0), hCursor: HCURSOR(0), hbrBackground: HBRUSH(0), lpszMenuName: PCWSTR(0x0), lpszClassName: PCWSTR(0x7ffbf3eff8e8), hIconSm: HICON(0) }
19:00:12 [DEBUG] (1) hudhook::hooks: HWND(788424)
19:00:12 [INFO] Output window: IDXGISwapChain(IUnknown(0x22b60745720))
19:00:12 [INFO] Desc: DXGI_SWAP_CHAIN_DESC { BufferDesc: DXGI_MODE_DESC { Width: 1920, Height: 1080, RefreshRate: DXGI_RATIONAL { Numerator: 0, Denominator: 1 }, Format: DXGI_FORMAT(28), ScanlineOrdering: DXGI_MODE_SCANLINE_ORDER(0), Scaling: DXGI_MODE_SCALING(0) }, SampleDesc: DXGI_SAMPLE_DESC { Count: 1, Quality: 0 }, BufferUsage: DXGI_USAGE(48), BufferCount: 2, OutputWindow: HWND(1771696), Windowed: BOOL(1), SwapEffect: DXGI_SWAP_EFFECT(3), Flags: 2114 }
19:00:25 [DEBUG] (2) hudhook::hooks::render: Could not lock in WndProc
...
19:01:01 [DEBUG] (2) hudhook::hooks::render: Could not lock in WndProc
19:01:01 [ERROR] Could not lock render engine

[veeenu]

@camas I stalked your GitHub and noticed you are using hudhook for Golf With Your Friends. I have that game so I can take a look at it directly! But yeah I can easily see how that's a synchronization issue. It could be solved by using threads and channels but I'm not sure I want the added overhead of those.

Weirdly enough, there is only 3 places now where the render engine is locked that could create contention:

• here during render
• here during wnd proc
• here during resize

My hunch there is that the wnd proc calls itself recursively, locks once and is not ever able to exit and do anything else. Or the WND_PROC static is initialized twice (which shouldn't happen as we use get_or_init that has the guarantee of uniqueness) and thus called recursively via CallWindowProcW.

[veeenu]

I have gotten some feedback and will now merge this in as there are no apparent regressions for the time being.

[vars1ty]

Thanks for the ping, feel free to ping me whenever as I'm more than happy to test out features whenever I can.

I'll update to the latest commit, fix a few errors due to the changes and then test it in both OpenGL and then later DirectX9, will let you know how it goes 👍🏽

[veeenu]

Thank you 🙏