Schwing getting started guide

A excellent getting started guide written by Schwing from here.

---

    ASM is hard. It's terrible in comparison to HLSL, which is similar to C++. There are times when 3DMigoto won't properly disassemble asm shaders, so there are HLSL errors. Most of the time they don't mean a difference. Other times, you'll need to remove the shader from ShaderFixes, even if it wasn't modified yet, in the case its presence in ShaderFixes breaks the relevant in-game graphics. Which means you'd have to work with the asm shader instead (i.e. not the "_replace" one). Even still, if you want to convert your manual fixes to regexes to match your solution to all shaders that have the same code (i.e. ShaderRegex, to support game updates), then you'll have to work in asm for that. That is why I hardcoded so much of Raft, as I don't have much time to work out the asm at this point.

    Continuing the information from earlier, here are all of the pieces of 3DMigoto I can think of:

• Fix steps and general information:
  • Pre-fix:
    • Extract the latest 3DMigoto release (v1.3.16 now) and keep it somewhere. Pick the x86 or x64 folder based on the game's bitness (32 or 64-bit), and stick d3d11.dll and the other files in the game's folder
    • Make sure the game works first while stereo is on and you have the settings you want to test with
    • Set verbose_overlay to 1 so you can see what shader is which during hunting. The warning doesn't seem to matter
    • Set export_shaders in the ini to 1 and export_fixed to 2 so you can dump the shaders into the ShaderCache folder. You may need to play past game launch to where the issue occurs in order to dump the shader in question. The game may even stall at launch and look like it's not responding until the dumping is complete. I don't keep it on unless I need it
    • Use the hotkeys for hunting to select and mark a shader. It appears in ShaderFixes so you can modify it manually
    • If possible, copy your game save into another slot periodically or to a different Windows folder so you can hop around the game (or use the game's level select if applicable)
    • Lastly, make a note of where you fixed an issue and at least a mental note of where other places use the effect. This will save having to play the whole game over again when you make a change
  • Fix:
    • Go through existing fixes and get a feel for what they should look like. You'll be working in d3dx.ini and any dumped shaders of interest. I write many comments and use complex solutions here and there, so start off easy with tools such as the below sections and work your way up from there. The other fixers have more experience fixing effects, so check multiple fixes for what types of formulas fix shadows for instance
    • The typical stereo correction formula is o0.x += flt_Separation * (o0.w - flt_Convergence); at the end of a vertex shader and a subtraction for a pixel shader. You may have to mess with the formula and variables, use a tweak ini param, or general guessing to determine what should fix something. The names I used above are from StereoParams and IniParams in shaders for separation (stereo.x), convergence (stereo.y), and the ini params that act as variables. Other shaderfixers usually use stereo.x, but you can call it separation if it's easier for you. The below sections help explain a lot of this, but you'll still need to go through fixes to get the full picture
    • The game engine defines how hard a game is to fix. So avoid a game engine if it's too difficult and come back to it. PCGamingWiki is useful for tracking this
    • More will make sense as you learn enough of 3DMigoto's tools, come back to old fixes, and practice
    • After fixing enough, modify the extracted release's ini to whatever values you want new fixes to use
    • When done fixing, double-check what you want is off, like hunting, dump_usage, and export_shaders. This is to save on performance by removing testing tools for your fix release
    • If you're C++ savvy, you can go into 3DMigoto's source code and look at how it works, what values the ini sections expect, etc.
    • If you know a scripting language like Python or AutoHotkey, you can create automations that make your life a lot easier and save a lot of time (I use FastKeys for many things). Manipulate files, text, etc. to perform several actions quickly or answer questions that aren't otherwise answered. This is the basis for DarkStarSword's asmtool.py that modified hundreds of shaders in his earlier fixes. It's still used to this day
• Ini params and local/global vars are variables for general use, for tracking what texture or menu (e.g. gameplay) is being used, etc.:
  • In my Raft fix, you'll notice there is a texture filter (x) and menu filter (z). X filters on textures found from frame analysis (explained later) in case I only want one graphic to be modified in shader code such as the crosshair in a HUD vertex shader. Z is used to make blanket fixes to a game's menu or gameplay screen (e.g. making a pause menu all one depth), or check what menu type to make a fix in
  • You can set values in the ini for use in the ini and in shaders. You cannot modify ini params in shaders, so it's a one-direction affair of working in d3dx.ini to match on texture hashes when needed in order to track what states you want (such as texture filtering) for when you check for it in a shader
  • E.g. x = ps-t0 means you want x to be what texture is being used for ps-t0. It will make more sense once you see the texture itself in Frame Analysis below, and filter on it with filter_index in [ShaderOverride]
  • In [Constants], specify global $var = 0.0 as my fix does if you want extra vars to save on ini param usage (where you don't use the var's value in a shader). You also need to have post $var = 0.0 in [Present] if you want to set it to 0.0 between frames so they start fresh before any ini sections are run. Then use the same post line at the end of each [ShaderOverride] it's used in if you want 0.0 between drawcalls to maintain the expected starting value (usually 0 to indicate "nothing"). Use local in a [ShaderOverride] if you want to create and burn a temporary variable
    • Special note: I highly recommend using vars to help only set ini params when necessary. This fixes performance in the event that there are so many times a shader is being used in a frame and x is being set to anything when it doesn't need to. My Raft fix also uses them to check if an ini param should be set back to 0.0 after a shader runs (i.e. post, see the below section for more information)
• [ShaderOverride]:
  • The main type of ini section where you do something when a shader is being run. Vertex and pixel shader combo, compute shader, or possibly multiple shaders at one time. Usually the former case. Domain and hull shaders are rarely important, although domain is used for the water layer in Raft
  • hash = deadbeef is required, so figure out what hash you want by hunting with the numpad keys mentioned in the ini
  • handling = skip is useful if you want to disable multiple shaders that get in the way of effects you want to look at. Otherwise use hunting to hide shaders if you're just looking for one. You can't have more than one hash in an [SO]
  • You can set ini params and perform actions:

    • [ShaderOverride_HUD_Crosshair]   
      hash = deadbeef   
      x      = ps-t0   
      
      [TextureOverride_Crosshair]
      hash            = deadbeef2
      filter_index = 1.0
      

    • This checks what texture is in ps-t0 and sets it to whatever [TO] is matching, and has either x = 1.0 or filter_index = 1.0. Note that filter_index ignores if statements, so use the former if you want to check for something before setting x. You use CheckTextureOverride = ps-t0 in the former case
    • You can use either pre or specify nothing for a prefix before a line if you want it to happen before the drawcall. Specify post instead for after the drawcall. I usually set ini params or global vars to 0.0 using post x = 0.0 to make sure the value is always the expected value in each subsequent drawcall
    • As I mentioned in the ini params section above, you can use vars here. My Raft fix uses $Temp_X to texture filter and check if it's > 0.0. If it is, then set x accordingly. If you don't want to performance test ini params, then make sure to use vars early in the fixing process
• [TextureOverride]:
  • Same as [SO], and you can also perform fuzzy matching:

    • [TextureOverride_Crosshair]
      ; hash = deadbeef2
      
      match_type        = Texture2D
      match_width       = height
      ; match_height    = 26
      match_bind_flags = shader_resource
      match_misc_flags = 0
      filter_index          = 1.0
      

    • This means I want a crosshair that is a square texture and is a shader resource without misc flags. You can dump a ShaderUsage.txt in frame anaysis to see what you need using the deadbeef2 hash you find from frame analysis. You may match multiple textures on accident or on purpose depending on your amount of testing
• Frame analysis (FA) is extremely important for troubleshooting:
  • You should uncomment the F8 line for analyse_frame. Then use a combination of these analyse_options values:
    • General:
      • Each frame (60 in a 60fps game) has 1 to n drawcalls that draws each graphic or does something until the nth one is the final image. The games I've fixed have always started with garbage, drew some objects, then eventually drew lighting before drawing the crosshair and HUD elements last. They are drawn on top of each other with the help of depth to tell what should be drawn
      • Render targets are what graphics are being drawn, depth targets are the depth from black to white of how far away to close an object being drawn is, and each drawcall has resources that FA can dump (textures, constant buffers, etc.)
      • Log.txt is helpful to read through the whole frame in case you don't have something dumped or want to see why it wasn't (e.g. see dump_cb above)
      • Set dump_usage to 1 in the ini to have ShaderUsage.txt appear, where it shows you the description of each texture. Go to the end and search backward for a texture hash. You'll see different match options (as explained for [TO]'s above)
      • Take the time to rename the FA folders to ..._Gameplay, ..._Pause or else you will easily lose track of them. I create many at once to compare either the logs or the dumped files. Permanently delete big folders you don't need with Shift+Delete
      • You can have either analyse_options globally (the existing ini line that is disabled), and/or create [ShaderOverride] sections that contain hash = and analyse_options = . Global options can be costly based on the options chosen, so try to bite less to chew on by using only [SO]'s for smaller scopes. If it's too much to chew, then feel free to hit F8 a second time to abort the frame analysis. It doesn't delete the incomplete FA folder
      • Examples of the options I typically use: analyse_options = dump_rt clear_rt, analyse_options = dump_rt dump_depth dump_tex clear_rt, or analyse_options = dump_rt dump_cb dump_vb txt clear_rt
    • Options:
      • dump_rt: you'll see the result of any drawing being done to a render target. Almost always use this so you keep track of where you are in the frame. For the first FA in a fix, always memorize the different stages the game goes through to understand the general structure for the game engine. Lastly, this is the least-costly option vs dump_tex and dump_depth, so go with this one when performing a global FA to get your bearings without it taking forever and eating disk space. Typically the game screen is drawn on o0 (considered the main render target)
      • dump_depth: costly to use globally, but any depth you need for adding dynamic crosshair depth will be at the end of a segment of the frame where it's being written. Look for it and wait until the last drawcall before it changes to something completely different (usually cleared to a blank one). It shows up as oD and there can only be one, but render targets can appear too. If depth isn't being written, it's being read in a texture slot instead (see dump_tex)
      • dump_tex: costly to use globally, but important for texture filtering. Typically the graphic being drawn is set to ps-t0. For instance, you see a crosshair graphic in ps-t0, so you know to use the hash next to ps-t0 in a [TextureOverride] section. I've seen as high as ps-t15, and in tracking a hash you may find that it was originally a render or depth target. For instance, depth being written to depth target oD can be read in a texture slot in a later drawcall as ps-t0, ps-t1, etc. (usually c505d7de is the main depth hash between games)
      • dump_cb: you can check out the analyse_options explanation in d3dx.ini, but I wanted to mention that you need to supply txt or buf in order to get these to show up in frame analysis. Eventually you'll want to see what constant buffers have in them, or match it (or something like a vertex buffer) by hash to help differentiate the drawcall from other ones
      • dds: if something doesn't dump, you can try dds. Usually it's depth that doesn't dump. You can specify jps to save on dumping time if you don't need DDS files as they have high time and storage space cost
      • hold: if an issue occurs once a second, you can either time the FA or supply this option and hold the F8 key instead. Don't forget that you can cancel an FA with a second F8 press
      • clear_rt: I always put this at the end to make sure the dumped files don't look wrong. IIRC it depends on the game for whether you need it
      • : you can search ^[^;]nalyse_opt in Notepad++ while in Regex search mode to find all active analyse_options lines. Disable them all with CTRL+Q to get just a log.txt if you want a really quick FA. You will more likely want dump_rt and clear_rt so you can see where you are in the frame
    • Opening working and corrupt dumped FA files:
      • Use the Nvidia 3D Vision Photo Viewer (nvstview.exe), XNViewMP, or vsgraphics.exe from Visual Studio to view jps and dds files. Most of the time, you'll see graphics you're interested in. You can use the Everything app from voidtools to easily find the exe files you need on your computer if they're installed
      • Sometimes, you'll get corrupted dds files in one of two cases:
        • Go into a hex editor like HxD to see if the 0x80th byte is 15. If it is, then replace it with a 14 and save it. It should work now
        • If it's not there, then you'll need to replace the header of the file. I don't have an example on-hand, but my notes say to replace the first 0x70 bytes with this 0x94 amount of bytes (so make sure it adds the remaining amount): 44 44 53 20 7C 00 00 00 0F 10 02 00 CC 03 00 00 80 0D 00 00 00 36 00 00 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 20 00 00 00 04 00 00 00 44 58 31 30 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 1D 00 00 00 03 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00
• Other ini sections and info
  • [CommandList] and [CustomShader]: look at my Raft fix to see how either is made. The former is a general ini section to run, and the latter lets you perform a Draw() or Dispatch() if it's a vs/ps combo or cs, respectively. Use a run = ... to have the section run from whatever ini section is running
  • [Key] and [CommandList] combo: my Raft fix sets up ini param y as a tweak value for testing so that I can hit Q and E to change y when I use it in a shader or the ini sections. Uncomment the two QE [Key] sections, one of the lines for both [CommandList]'s, and add ... = ... * flt_Tweak_Value in shader code. This serves as a tool for scaling a formula by keypress, or flip something on/off while you test something in-game
  • Use the performance monitor (F9 lines in the ini) to cycle through what's causing bad performance, and check what the fps is. You can see what ini sections are worst and the drawcall overhead among other things
  • If a game is stuttering or stalling, you can enable cache_shaders to create bin files that prevent it from happening a second time
  • I use Nvidia Nsight to debug performance. You'll have to rename 3DMigoto's d3d11.dll and guess what shader is being used when clicking Disassembly after choosing a drawcall. You can sort the events by CPU and GPU usage in order to tell what's worst-performing and even see resources similar to how FA works
  • If you're testing a new game and 3DMigoto doesn't work in some way, enable calls and possibly debug in [Logging] to dump out d3d11_log.txt and see what errors show up
  • Mess with the other ini settings like in [System] if you need the game to boot at all. Check PCGamingWiki to confirm it's a DX11 game. Use DGVoodoo if it's a DX9 game and you want to convert it to DX11 for 3DMigoto to work
  • Memorize the hotkeys so you have more options to work with, like the mode select for hunting (e.g. mono, skip, pink, and mono). Hit + so you can stop hunting, and hit F10 to reload any changes you make. Eventually you'll come across a bug where changing an hlsl file in ShaderFixes doesn't affect F10. Change any shader (modify the comments) and hit F10 and it should reload your changes
  • If you know there are compute shaders in the game, you'll likely need to use StereoFlagsDX10 = 0x4008 or 0x5008 (masterotaku uses that) in [Profile] to get one-eye lighting to be stereo. Also check fixes to see what they use for the other [Profile] settings

    If this feels like a lot, then that's why the interactive learning tool would be better. It's a week-long class worth of information to explain every little thing that would be easier to learn in an interactive environment. If time permits this year, I'd like to implement it at some point. For now, this will get you going better than I did starting out.