Special K: Experimentally re-enable support for ReShade and Special K

[sonic2kk]

If this works, it will resolve #894.

Previously ReShade and SpecialK had a crash conflict. This may no longer be the case. Special K has what appears to be a different way of handling ReShade DLLs now which may be more compatible under Wine.

The code in STL copies the ReShade DLLs over into the game exe folder without renaming them, as SpecialK is injected into the game with dxgi.dll (or another architecture, possibly auto-detected based on game API in the game list HTML file?) so it is responsible for picking up the ReShade32.dll/ReShade64.dll files and injecting them.

The settings for the game exe folder is taken from the same one as the ReShade setting, and as well the logic for this expects ReShade and SpecialK to be both enabled. However, because the code for this ReShade+Special K integration was written long ago, it doesn't include the ReShade Version Override changes from #649, and also probably does not include the ReShade automatic update. Both of these are potential future improvements to the integration.

We should also make sure the ReShade DLL naming from #881 does not apply when working with Special K (ex: if USESPECIALK, don't use the ReShade DLL name code). From glancing at the code this should be the case, but I have not tested.

The code for 99% of this was written by Frostworx long ago and commented out, when it was found that ReShade and SpecialK caused games to crash if used together. Only minor refactoring has been done to account for some changes since this code was commented out (such as #696 impacting a couple of lines).

---

This code is untested by me for now. Another user has volunteered to test and see what cosmic horrors await by re-enabling this. Let us hope for success 🫡

[sonic2kk]

See discussion in #894, but essentially the following is needed:

1. Move un-renamed ReShade DLLs into same folder as game exe
2. Make sure ReShade INI, if it exists, uses correct line-endings
3. If it doesn't exist, create it to stop ReShade from creating its own INI when launched with SpecialK, causing a game crash (we can't modify the line endings as this is created during game launch, so STL can't do anything at this point)
4. Conditionally use d3dcompiler_47 -- It is required when using Proton Experimental, but breaks when using GE-Proton8-14.
   • In other words, ONLY export the d3dcompiler_47 as an override if we're using ReShade+SpecialK with Valve Proton (Proton-tkg etc are untested but I imagine it behaves like GE since both use the same gst plugins by default afaik?)
   • d3dcompiler_47 is only required for ReShade standalone regardless of Proton version, but it is not required at all when using SpecialK
   • The actual conflict with d3dcompiler_47 is that GE-Proton crashes when trying to load some SpecialK Sound assets if this DLL is used, but Valve Proton is the opposite. See ReShade compatibility with Special K #894 (comment) for more details.

We don't need to put the DLLs into the game prefix in a PlugIns folder, or modify any INI files. Everything should go into the game folder.

We need to know how to handle various ReShade options when using SpecialK as well, namely:

• Using GE-Proton, where d3dcompiler_47 is disabled, we need to properly handle disabling it
• We need to make sure we handle ReShade being installed BEFORE SpecialK -- In other words, how do we handle migrating an existing ReShade installation to a SpecialK one? Especially when it comes to DLL name conflicts like dxgi.dll
  • We should disable the DLL naming entirely if it isn't already when using SpecialK+ReShade, and we should document this on the wiki
• Probably other constellations too

[sonic2kk]

In the case of NieR:Automata (a 64bit D3D11 game), renaming the SpecialK DLL from dxgi.dll to d3d11.dll seems to fix the issue where the SpecialK UI will activate but be invisible. Sadly the ReShade.ini created here is different than before with the dxgi.dll and some of the options can cause crashing. This is inconsistent and I haven't narrowed it down yet (mainly because I can't re-create the problem at all now...) but this is worth bearing in mind.

The DLL name used by SpecialK appears to be important here.

And also, all of this is manual for now (except for the SpecialK installation). We need to know the behaviour and parameters first before we attempt an implementation. What is clear, though, is that this PR needs more work to accommodate this.

[sonic2kk]

Pushed a hefty commit that does a few things:

• Copies all ReShade DLLs and JSON files into the game folder, instead of trying to use the SpecialK folder in the prefix
• Skips renaming the ReShade DLLs when we're using SpecialK+ReShade
• Uses installRSdll to move the files, which in theory could help with implementing ReShade Update logic
• Remove d3dcompiler_47 installation for SpecialK standalone
• Tested installing ReShade on top of an existing SpecialK installation, worked fine (as expected)

There is still some work left to do:

• The DLL namings are not correct automatically, I wonder if we need to change how we fetch the graphics API and how we map that to the SpeicalK DLL name?
• No conditional logic for when to copy d3dcompiler_47, so if using Proton Experimental, things won't work
• No testing on disabling ReShade with SpecialK still enabled
• No testing of disabling SpecialK with ReShade enabled
• No testing of installing SpecialK on top of an existing ReShade installation
  • Ideal behaviour for this scenario would be to entirely remove the existing ReShade installation, and replace it with the SpecialK installation
• No logic to force dos2unix to fix invalid potential ReShade.ini line endings issues

---

With this PR in its current state, testing with NieR:Automata with GE-Proton8-14 using SteamTinkerLaunch with SpecialK and ReShade enabled, and then renaming dxgi.dll/dxgi.ini to d3d11.dll, and finally renaming the DLL name in SpecialK_enabled.txt to use d3d11.dll as well, SpecialK and NieR:Automata appear to consistently work together!

[sonic2kk]

Tested NieR:Replicant, it needed the DLL rename from dxgi.dll to d3d11.dll -- I wonder if it's worth having a checkbox for this option.

Interestingly, it seems now neither game needs d3dcompiler_47? Very strange indeed.

[sonic2kk]

As of right now, for NieR:Automata, SteamTinkerLaunch, SpecialK and ReShade can be used together with no manual intervention required! The DLLs will be copied over and SpecialK will load ReShade as normal.

NieR:Replicant still fails because it fails to find the rendering API, and falls back to dxgi (intended behaviour), though it should find Direct3D 11 and correctly use d3d11.dll.

Intended behaviour is currently to use d3d11.dll if we know the API is Direct3D 11, as in the two games tested, this allowed SpecialK to load correctly, as well as for it to load correctly with ReShade -- In other words, it is a seemingly safer choice than dxgi. However, if we can't find the API (perhaps DX10?) we will fall back to dxgi.dll to be safe.

I also noticed that we don't check for d3d12 at all, I wonder if it can use dxgi? I think it can, I believe this is what ReShade uses for DX12.

Once the check here is improved so that Direct3D 11 titles can be found, NieR:Replicant should work.

Missing features:

• Auto-updating ReShade (needs testing)
• Re-Shade version override (needs testing)
• Disabling SpecialK and ReShade (when SpecialK is off, the ReShade DLLs will simply not be loaded, maybe we don't have to do anything here, though if they're both off maybe we should remove them? So games uninstall more cleanly)
• Installing SpecialK on top of an existing ReShade installation (all existing tracked ReShade file should be purged and ReShade should be re-installed)

[sonic2kk]

I changed the logic for finding the SpecialK rendering API to be the following:

• If DX12, use dxgi,dll
• If DX11, use d3d11.dll (this may not be compatible with all games, requires more testing!)
• If DX9, use d3d9.dll
• If game is found but rendering API is unknown, use dxgi.dll
• If game is not found, use d3d11.dll (not ideal, because there could be a d3d9 game that is not found and so will not work)

In testing with NieR:Automata and NieR:Replicant, d3d11.dll seems to be required for using SpecialK and ReShade together. It also seems to work when just using SpecialK standalone.

[sonic2kk]

ReShade Override and ReShade Update work flawlessly.

Sadly it was discovered that for NieR:Automata and NieR:Replicant, only ReShade 5.4.2 and below(?) appear to work. Newer versions introduce a conflict with ReShade seems to want to load over SpecialK, causing SpecialK's OSD not to appear, and its menu to be invisible.

One solution to this is the following:

1. Add a SpecialK ReShade version, defaulted to 5.4.2
2. Add a checkbox under ReShade settings that says "Use ReShade Override with SpecialK" - similar to the ReShade with custom commands checkbox

This way, if a game does support a newer ReShade version in combination with SpecialK, they can override the version for that game. Similarly, if support does improve across the board for the ReShade version which can be used with SpecialK, then a user can bump this. We can also bump this too in updates, so we control which version of ReShade is used with SpecialK.

We would also, of course, have to document this on the wiki.

[sonic2kk]

With the PR in its current state, NieR:Automata and NieR:Replicant both work with ReShade and SpecialK provided the ReShade version used is 5.4.2. This may vary from game to game, and in future the default will be ReShade 5.4.2 so that the user will not have to specify this, with an option to set a global SpecialK ReShade version should compatibility improve.

Version is bumped, langfiles are unchanged, shellcheck is green, so this PR is ready to merge! One step closer towards ReShade and SpecialK playing nice together.