Salmanko's Analysis of the Security System and Mismatch Mechanism in Generals: Zero Hour (GeneralsMD)

[lostsquirrel1]

Analysis of the Security System and Mismatch Mechanism in Generals: Zero Hour (GeneralsMD)

Cheat Detection in GeneralsMD

Generals: Zero Hour runs on a deterministic lockstep engine, meaning every computer runs exactly the same game events. If there's even a tiny difference in the game state between players, it's immediately caught as a "mismatch."

There isn't a separate anti-cheat system in the code that scans for cheaters or instantly bans them. Instead, the game heavily relies on checking for any differences in the game state as its main security measure. In the source code, you can see a structure for verifying state integrity using a CRC (Cyclic Redundancy Check). For example, the GameLogic class has variables like m_CRC and m_shouldValidateCRCs to keep track of the CRC and its validation status.

When you play online, the CRC check is turned on. Every machine periodically calculates a checksum representing the game state (like unit positions, health, resources, etc.) and compares it with everyone else's values. If any machine's checksum doesn't match, the game figures that something's off, shows a "Game has detected a mismatch" message, and ends the match.

That error message basically means the game state data isn't identical across all players. The source code even includes special files for this, like CRCDebug.h, XferCRC.h, and XferDeepCRC.h, which show that there's a system in place to deeply compute the CRC over all game data. Even though you don't see an obvious spot in the code where m_shouldValidateCRCs is set to TRUE, it's most likely activated when a multiplayer match starts. The whole idea is to ensure every tiny detail of the game stays in sync—any variation, whether from cheating or minor calculation quirks, gets caught immediately.

A technical tidbit: the developers mentioned that differences in CPU settings (like enabling SSE2) can sometimes cause tiny calculation differences that end up triggering a mismatch. This shows just how sensitive the engine is to even the smallest discrepancies.

Keeping Game Data in Sync

The game's core idea is all about strict synchronization. Players send each other commands (like building or attacking), and every copy of the game executes the same command at the exact same moment. The game doesn't send the entire "game state" over the network—only the commands are shared. This means everyone needs to have the same initial state and data files to get the same results. For example, the code includes files like MultiplayerSettings.h and likely has some settings to ensure everyone is on the same page.

There's also a module called CopyProtection in the code, which might be used to verify that everyone's using the same version of the game and hasn't messed with the main files (like checking the game disc or ensuring core files haven't been altered). In practice, if one player tries to use different data files (say, by modifying a unit file or using a custom map.ini that others don't have), it usually results in a mismatch as soon as the game starts or during play. For example, if someone tweaks the map.ini for a custom map but the other players don't have the same tweaks, the mismatch shows up immediately. So, the game makes sure that the content is exactly the same for everyone both before and during the match—any difference in data or game version will trigger a mismatch.

On top of that, the game engine also logically validates every command. For example, when a command is sent to build a unit or structure, the code checks if the player actually has enough resources and meets all the requirements. This check happens on every machine, not just the one that sent the command. So if a player cheats by boosting their money locally (using a cheat tool or memory hack) and then sends a command to build an expensive unit they couldn't normally afford, things go haywire: on the cheater's machine the command goes through because of the fake money, while on the other players' machines the command either gets rejected (because they don't have enough real money) or is processed in a totally different way. This leads to one machine creating the unit while the others don't—or they handle the resources differently—which causes an immediate mismatch in that game frame.

In short, any tampering with internal values (money, unit health, POWER, etc.) on one machine without the others doing the same will cause that player's game simulation to drift away from everyone else's, and the CRC system catches this instantly, ending the match to keep things fair.

How Synchronization Works and How Mismatches Happen When Cheating

A mismatch happens when the synchronization mechanism fails—that is, when the game's "world state" isn't identical across all devices. Every player's computer runs its own state machine that represents everything happening in the game (units, positions, health, money, research, and so on). All these states start off exactly the same at the beginning of a match and only change when commands (actions) are executed that are shared over the network. The system assumes that as long as the inputs are identical and the engine behaves the same, the game states should remain in sync. If something goes off, it means there's either a cheat or a bug. The code likely calculates a periodic CRC of the game state (maybe every few frames or at specific events) and sends it between players or to the host. If any CRCs don't match, the game immediately ends with an error message.

There's also a hint from the developer community that enabling certain instruction sets (like SSE2) might help prevent mismatches caused by floating-point calculation errors. This suggests that technical issues causing mismatches might not only be from intentional cheating, but also from tiny differences in how different processors or settings (like the old x87 FPU mode versus SSE2) handle calculations. That's why the engine is designed to be as deterministic as possible. For example, traditional games use a shared "seed" for random number generation, so every random event is based on that same seed to ensure consistency. Any deviation—like one machine producing a different random value or order—will break the synchronization.

Reasons for Mismatches and Theoretical Ways to Bypass the System

Technically speaking, mismatches boil down to two main reasons:

1. Differences in the initial files or data among players. This covers scenarios like not everyone having the same game version or patch, or even minor differences in unit or map files (for example, one player might have a modified file that makes a certain tank stronger). Such differences show up as soon as that element is used in the game.
2. Differences in events or outcomes during the game. This includes in-game cheating scenarios, such as memory editing (altering resources, health, etc.) or sending illegal commands. In both cases, the periodic sync check catches the discrepancy.

There was also a known case where some players intentionally tried to force a mismatch as a "weapon" — deliberately causing a mismatch to end the match without recording a loss (for instance, to avoid a defeat on their record). They'd do this by triggering an impossible game state through on-the-fly cheating (like building a structure that doesn't exist in everyone else's files, or sending bogus network commands). The system would spot it and shut the game down; while it wouldn't give the cheater a win, it also wouldn't record a loss (the match just gets canceled). So, this loophole was sometimes exploited, even though it didn't really give the cheater a direct gameplay advantage—it was just an underhanded tactic to avoid a loss.

Is it possible to bypass this detection system? Practically speaking, it's really hard unless all players are in on it. The power of the sync system is that it doesn't allow one machine to go off on its own; either all machines go off together (so no mismatch is detected) or the game ends. The only way to cheat without getting caught is to keep the cheating synchronized across all clients. For example, if every player (whether by agreement or trickery) uses the same modifications or files, then there won't be any differences in the calculated state—and the engine won't notice anything because everyone's doing the same shady stuff. Of course, this scenario only works if everyone is colluding.

Now, imagine a custom, "modified" map where the creator inserts code that gives them a secret advantage (like a hidden super unit). If all the other players end up playing on that same modified map without knowing it, then they're all basically following the same game rules (even if those rules are unbalanced), and no mismatch occurs because the engine sees everything as normal and in sync. This is more like fooling the game itself rather than outright breaking it. It's an unethical way to cheat, but it stays within the boundaries of the synchronization system—essentially cheating "by the rules" instead of breaking them—so it doesn't get flagged as a technical violation.

On the other hand, trying to bypass the system by modifying the code on one machine while leaving everyone else's untouched is nearly impossible without causing a mismatch, because at least one other machine will spot the CRC difference. There's even the theoretical idea of patching the game engine to disable the CRC check or ignore mismatches, but that isn't practical either. Even if you stop the mismatch alert on your own machine, the other players would notice and shut the match down, or your game would run out of sync on its own. Similarly, changing the calculation method to output the same CRC despite differences would require full knowledge of everyone else's game state—meaning the cheater would have to calculate "what the expected CRC is for the others" and then fake their data to match it. This isn't really doable unless you control all devices or break the communication encryption, if there is any.

So, the game's security is built on a simple yet effective principle: either everyone plays by the same rules and stays synchronized, or the game ends for everyone.

In a Nutshell

The GeneralsMD code includes cheat detection systems that ensure total synchronization among players. Any break in that sync—whether from cheating or a bug—results in a mismatch and ends the match. The only theoretical ways to get around it are either to cheat on all machines simultaneously (which isn't really a one-sided cheat) or to use cheats that don't affect the distributed game state. The latter includes things like map hacks that remove the fog of war for a player locally without changing any in-game objects—this kind of cheat gives you extra information but doesn't alter unit or resource states, so it doesn't trigger a mismatch (since what you see locally isn't part of the engine's calculations). Similarly, tweaks to the user interface or camera that let you see further don't impact synchronization. These methods are possible because GeneralsMD's security focuses on the core game state rather than what's shown on your screen. Still, they're much less impactful compared to direct cheats on resources or units, which the synchronization system quickly catches.

Source Code References

GameLogic.cpp
GameLogicDispatch.cpp
CrateSystem.cpp
PerfTimer.cpp
GameClient.cpp

((important note ))
There may be errors in expression or explanation. My friend Ak69 reviewed the text to make sure the meanings reached the reader well, and I apologize if there was no clarity in understanding or other terms. As for the analysis, it is early, meaning that the analysis was based on previous old experiences and new experiences with the source code. This is still the first part, and there are 3 other parts left, and two other practical experiences in a real match. I hope you will correct any mistakes for me.

Programs used in the experiment:

---

Relcass
x86dbg
OllyDbg
PE-bear
There are also programs I created that do not have a name -------- > ( 3 )

[lostsquirrel1]

Tarnished Knight here.

Thanks for putting this together, Salmanko. It is nicely put together. I've read through it and would like to share my thoughts with a view on enhancing this already good document.

1. You say the tiny differences in the simulation is caught immediately by the mismatch CRC check. For relative definitions of immediate, that is true, for example: from a player's perspective. However, it is likely that it take several simulation frames before the difference is caught (may even a few seconds in some cases.) You admit that indirectly we you later say: "Every machine periodically calculates a
   checksum". Perhaps it would be better to explain that the mismatch is caught shortly after the issue (this will also help hint to programmers why desyncs can be some of the hardest bugs to fix.)

2. I notice several references to the effect of SSE vs FPU. I think there is some ambiguity that will cause confusion here. FPU and SSE instructions are different instructions, so if a program is compiled using FPU instructions, all machines will use the FPU to calculate the results, not the SSE. I doesn't matter what the capabilities of x86-based processor are.

Side Note: If the program has CPU-dispatch functions then this could be the case, but I don't believe that is something we need to contend with here. CPU-dispatch functions have to be custom written and require a lot of technical knowledge and have quite an impact on the code. It also wouldn't be in the interest of the deterministic simulation to do so.

3. I feel there's perhaps an over emphasis on the anti-cheat nature of lockstep networking. Lockstep is chosen for RTS games primarily because it allows for the handling of hundreds or thousands of active entities without having to contend with the constraints of network bandwidth. The anti-cheat aspect of it is more a useful side effect - as is providing an easy system for handling replays.

[Chikinsupu]

Tarnished Knight, you seem to understand the concept of lockstepping and determinism quite well, so first of all thank you for clarifying this:

it allows for the handling of hundreds or thousands of active entities without having to contend with the constraints of network bandwidth. The anti-cheat aspect of it is more a useful side effect

That being said, what is your opinion on utilizing fixed point math? I worked on an online RTS as my last student project some time ago and we had created our own fixed point class to guarantee determinism. It was all written in C# so probably not of much use in this project, but hey.
Do you think that would also be an option to consider or might that be too overkill to replace a ton of floating point math with fixed point math now?

[lostsquirrel1]

Hi there. Happy to hear my wall of text was useful to someone! :)

As I understand it, Commercial RTS games used fixed point math in the past typically for two reasons. 1) before developers figured out how to make floating point deterministic - there was a general opinion for a long time that it wasn't possible, even when several had managed it. 2) multi-architecture support: it is easier to get ARM, PPC, x86, and SPARC to all behave the same with fixed point math and some of these architectures didn't have a IEEE754 compliant FPU for a very long time.

I think it is a fine approach if you want to avoid the complexities (and surprises) of floating-point determinism.

For a project like this. We have a lot of smart people to hand, and the intricacies of floating-point determinism are now generally well understood and, as you pointed out, the engine already is using floating point math. If we can take on the challenge (and I think we can), it would be in our best interest to be using SSE for performance reasons. There is a library that allows us to maintain bit compatibility with SSE using arm64's NEON. So with a bit of work, we can achieve cross-platform syncing without needing to fall back to fixed-point math. Though, if something unexpected does come up with the floating-point approach, then we know that we can always fall back on fixed-point math to keep arm64<->x64 syncing.

[xezon]

Very nice explanations. Well done!

[OmniBlade]

No need to worry about SSE>NEON libraries, those exist to make code using SSE intrinsics portable to ARM. The main thing to care about is what FLT_EVAL_METHOD is set to which tells you what the precision intermediate calculations are made in. If this matches between platforms, then the basic operations and sqrt should produce the same results at a binary level.

The difficulty comes with all those trancendental functions sine, cosine and friends, those do not have standardised output beyond they must be correctly rounded to a certain level of precision which gives wiggle room for the exact values returned. This means that they can return slightly different results depending on how the math library implements them and if it uses a CPU instruction for it or not. Not fine if you are checksuming the results at a binary level and using them to decide if you are in sync with another version of the simulation. The solution there is to ship your own implementations of those functions so the same basic operations are used on all CPUs to get to the result.

[augustresende]

There is some 'we must look' codes:

---

https://github.com/electronicarts/CnC_Generals_Zero_Hour/blob/0a05454d8574207440a5fb15241b98ad0b435590/Generals/Code/GameEngine/Source/GameLogic/System/GameLogic.cpp#L187C1-L208C1

"what if other part of the game's code change from the default setFPMode() to another mode while the game is trying to calculate the state's CRC?"

This code is altering the FPU’s behavior in ways that can easily lead to subtle differences in numerical results between platforms or even runs. Here are some issues it can cause:

• Rounding Mode Differences:
  Although the comment mentions “chop” (truncation), the code currently sets the rounding mode to “round-to-nearest.” If parts of the game’s logic expect truncation or if different modules assume different rounding modes, even slight differences in rounding can accumulate over many calculations, leading to mismatches.

• Precision Control Effects:
  By forcing a low precision (24-bit) mode, the code reduces the accuracy of floating point operations. On platforms or CPUs with different default precision settings (like modern SSE units on Linux/macOS compared to the legacy x87 FPU on older Windows systems), the results can diverge, especially in iterative calculations or physics simulations.

• Platform and Library Differences:
  Under Proton/Wine on Linux or macOS, the emulation layers may not replicate the exact behavior of the original Windows FPU settings. Differences in how these platforms handle the FPU control word or default to using SIMD instructions can cause consistent mismatches.

• Determinism in Networked Games:
  In multiplayer scenarios, even tiny differences in floating point math can lead to desynchronization. If one player's system computes values slightly differently due to the FPU settings or hardware differences, then the game state can diverge, which might be what you’re observing.

• Legacy Code Assumptions:
  The game, being from 2003, likely assumes a very specific behavior of the floating point unit based on the hardware and compiler of that time. Modern systems, with different floating point units, optimizations, and even different default FPU states, can break those assumptions.

In summary, the forced settings can lead to non-deterministic behavior across different systems. This is likely why you see mismatches in calculations among players and consistent discrepancies when using Proton/Wine. Addressing these issues might require ensuring consistent floating point settings across all environments or revisiting the game’s reliance on precise floating point determinism.

---

https://github.com/electronicarts/CnC_Generals_Zero_Hour/blob/0a05454d8574207440a5fb15241b98ad0b435590/Generals/Code/GameEngine/Include/GameLogic/FPUControl.h#L35C1-L39C4

DirectX can mess with FPU configuration

  * setFPMode sets the FPU internal precision and rounding mode.  As DirectX is not guaranteed to
    * leave the FPU in a good state, we must call this at the start of GameLogic::update() and
    * anywhere that touches DirectX inside GameLogic loops (LoadScreen).
    */ 

About that directx fpu messing apparently this can be prevented too

https://irrlicht.sourceforge.io/forum/viewtopic.php?t=8773

D3DCREATE_FPU_PRESERVE

---

https://github.com/electronicarts/CnC_Generals_Zero_Hour/blob/0a05454d8574207440a5fb15241b98ad0b435590/Generals/Code/GameEngine/Source/Common/RandomValue.cpp#L54

theMultFactor can be one of the mismatch causes

For 32‐bit `UnsignedInt`, that’s `1.0f / 4294967295.0f`. In **single precision**, the resulting float might be **very close** to ~2.3283064e-10, but the exact rounding could differ slightly on different architectures or if the FPU is set to a different precision mode (`_PC_24` vs. 80‐bit x87 vs. SSE2 vs. ARM NEON emulation).  

If two machines do **not** produce bitwise‐identical results for `((float)RANDOM * theMultFactor)`, the final `(delta * ... )` can differ in the last bit. Over time, these tiny divergences can yield different object positions or ballistic outcomes → diverging game states → a **CRC mismatch**.

### Why is that subtle?  
- On older x86 hardware with the same `_controlfp()` settings, all players got the same rounding.  
- On Apple Silicon or modern x64 SSE, the result may differ by 1 ULP (unit in the last place).  

So **yes**, the `Real` random calls **can** cause mismatches if the FP environment is not forced identical.

"what uses a lot of GameLogicRandomValueReal? AI"

---

some maybe related URLs:

llvm/llvm-project#44218
lifting-bits/remill#178
86Box/86Box#507

can softfloat be an alternative? needs testing
https://github.com/InfiniBrains/softfloat

[nhatter]

If two machines do not produce bitwise‐identical results for ((float)RANDOM * theMultFactor), the final (delta * ... ) can differ in the last bit. Over time, these tiny divergences can yield different object positions or ballistic outcomes → diverging game states → a CRC mismatch.

Why is that subtle?

• On older x86 hardware with the same _controlfp() settings, all players got the same rounding.
• On Apple Silicon or modern x64 SSE, the result may differ by 1 ULP (unit in the last place).

So yes, the Real random calls can cause mismatches if the FP environment is not forced identical.

Attached an instant mismatch from a 1v1 game (happens about 9 seconds in). Both running on same network, with v1.04 and Gentool.

Client 1: Windows 10 (Parallels Desktop VM on Intel-based 2014 Macbook, v1.04 + Gentool)
Client 2: Windows 11 (Parallels Desktop VM on Silicon-based M4 Macbook, v1.04 + Gentool)

Might be helpful for debugging.

Instant Mismatch.zip

[lostsquirrel1]

This may also be a useful source of information from the Recoil Engine: https://github.com/beyond-all-reason/spring/wiki/Determinism-In-Engine

[OmniBlade]

Of the issues listed there, the random calls as nested function calls is only an issue between compilers. It won't be causing the existing desyncs we have in SAGE because its only one binary, but could be an issue in the future for cross platform sync. I experienced that myself trying to get cross platform networking to work on Vanilla Conquer.

[OmniBlade]

Another consideration we need is floating point contraction, so we will likely need -ffp-contract=off for clang/gcc, /fp:strict for msvc < 2022 and /fp:precise for >= 2022.

[CarlosR759]

With this anti cheat model someone can be able to fetch match data without getting a game hash mismatch ? Because if you only read the memory from the game but not alter the game stack of user instructions, you could in theory play the game and have total knowledge of units positions and total resources of every player. Probably back in the time with one core cpu that could be hard to do without getting caught. But now probably you could just set the game to use specific cores of the cpu and fetch the game data from other cpu cores.

[Fighter19]

I consider @augustresende 's idea about using softfloat one of the best for instant remedy.
GCC appears to already have a softfloat implementation.
Check out this stackoverflow to see how to use on x86.
Current PCs are beefy enough to handle softfloats. And if required we could even adjust it to replicate the old behavior.
I tried using "-msoft-float" on GCC 12 and the generated assembly looks just fine, without any FPU instructions.

Also the Xfer routine seems to use the pure value stored in the float.
So even a float of 0x00fbdb46 vs 0x00fbdb45 would result in the synchronization failing.
Even though it's such a small difference, that no one would notice and would likely disappear in any operation, with a float of a higher exponent.

Thus, I currently don't even see the "error accumulation" as the likely reason for the desyncs.

My personal opinion, is to take one of these approaches, if breaking compat is considered acceptable:

• Get rid of floats in GameLogic (preferred as it's the easiest to understand, leads to faster code and is definitely cross-platform)
• Quantize the floats (I believe this is what they tried with the 24-bit precision, although it pretty much went around to bite us, it appears)
• Read the C++ documentation and try to figure out if "std::roundf" and co. are well defined. (according to cppreference it should always round the same. To my knowledge that doesn't mean however that it will always return the same across different platforms)
• Ensure, that the precision is decreased before calculating CRC (this will not fix it, but drastically reduce desyncs)

Last step should be easily testable by excluding LSBs of the mantissa.

void Xfer::xferReal( Real *realData )
{
	uint32_t realDataInt = *(uint32_t*)realData;
	realDataInt >> 13;

	xferImplementation( &realDataInt, sizeof( Real ) ); 
	
}  // end xferReal

• Or as someone mentioned just send a snapshot of all floats from the lobby master to the clients in certain periods or if desyncs occur.
  (This is how it's done in the Overwatch netcode and is likely what Unreal is doing as well).

Personally, I wouldn't bother with trying to get the floats calculate exactly the same on each client.

[Fighter19]

Oh and one more thing I'd like to mention, is that the problem with DirectX accessing float mode is likely a result of using D3DXMatrix.
So I believe it's a D3DX problem, not a D3D problem.
By providing a custom implementation (look at my repo to see my reimplementation),
the DirectX problem mentioned in the source code likely won't occur.

This should be easily verifiable by running the original game with DXVK (as D3D8) under Windows and seeing if it easily desyncs.
If the source code is actually right (which I doubt) and the problem is really in D3D, desyncs should occur as a result.

[nhatter]

This seems like a sound theory of why mismatch is happening.... indeed, perhaps using non-floats (eg. longs) might be faster and more cross-platform. Still getting instant mismatch on Silicon/ARM64-based Mac (with Windows 11) when playing online via Direct Connect (with GameRanger). Most likely due to subtle differences in float calculation on ARM64 - see the attached replay file from ZH - hope this helps with testing / investigating: Instant Mismatch.zip

[OmniBlade]

Using longs isn't an option for replacing floating point operations in general and would be a massive undertaking to replace all the float maths even just in game logic I imagine. Floating point operations are heavily optimised on modern CPUs so you are also throwing away performance by not using them. I'd want to make sure we can't get deterministic floating point calculations cross platform before worrying about invasive changes.

[Fighter19]

If you can find a way to make them consistent, sure.
But even then, we can compile with -msoftfloat to instrument the generated code with calls to functions instead of floating point instructions,
then write our own implementations for these functions. They could happen to run SSE or NEON code and apply consistency fixups on the operations that cause problems. That would require almost no changes to existing code.

I don't know enough about ARM to know exactly what's causing the problem.
Any hopefully @nhatter 's replay can be used to determine where the differences occur.
I can almost bet, that it has to do with 24-bit precision, though.

If you look at Wine's implementation for _controlfp, you can see, that any handling of "PC_24" is missing for ARM and the likes.
I also can't find any reference to low-precision or 24-bit precision in ARM's NEON or VFPU manuals.
This is likely, why it desyncs immediately on ARM devices.

So for testing, one could deactivate setFPMode and the fast_float variants (something I think is already done)
then try running it. If float is not completely doomed, then single-precision floats should be working just fine.

And if we NEED 1.04 compatibility, then I don't see a way around using wrappers, to emulate 24-bit precision.
I believe it's not even implemented on all CPUs anymore and quietly uses 32-bit precision, but that's just a theory.

[OmniBlade]

Based on some playing around in godbolt, I believe that its feasible to get binary matching float behaviour between 64bit builds using SSE and preventing contraction using the FMA instructions. 32bit I'm less convinced because due to the ABI using the x87 to return values MSVC still chooses to use the x87 for some calculations even if SSE is enabled. 32bit mingw could be made to work though. We'll still need to provide our own maths library to avoid differences in implementation for trancedental functions.

What we can't do is get binary matching with the original x87 FPU instructions, but I don't think we should have ambition to do so beyond the vc6 builds.

[nhatter]

@augustresende @Fighter19 I just spoke to Kris Morness (former Generals + Zero Hour engineer) on LinkedIn and he said:

"Yep, wrapping it through an API like [softfloat] could solve it. I also can't prove this but a lot of the methods we used to keep floats in sync might no longer be helpful. That wasn't something I worked on back then but when I worked on [different famous RTS game], there was some old assembly code that caused issues and I ripped it all out. Some people still think you can get floating point drift on modern systems but also most people don't make P2P lockstep anymore."

Hope this helps. Oh, and he's VERY generously offered to join us in Discord to help answering questions! I'll send him the link for Community Outpost Discord ASAP... wow!

[xezon]

nice

[helmutbuhler]

Technically speaking, mismatches boil down to two main reasons:

1. Differences in the initial files or data among players. This covers scenarios like not everyone having the same game version or patch, or even minor differences in unit or map files (for example, one player might have a modified file that makes a certain tank stronger). Such differences show up as soon as that element is used in the game.

2. Differences in events or outcomes during the game. This includes in-game cheating scenarios, such as memory editing (altering resources, health, etc.) or sending illegal commands. In both cases, the periodic sync check catches the discrepancy.

Just to expand on this because this hasn't been mentioned here yet. There are more possibilities as to how mismatches can occur:

• Non-Determinism: If the update code doesn't always result in a determinate next state, mismatches can occur. This can happen for example if you run into undefined C++ behaviour (non initialized variables, out of bounds memory access, etc). It can also happen with trivial things like sorting a list by pointer, and ASLR randomizes your addresses.
• Memory-Corruption can also cause mismatches.
• The initial gamestate can also depend on previous games, if the game wasn't restarted inbetween. This also doesn't necessarily result in a mismatch directly after starting the game. It's possible that some of these not-correctly-initialized variables are not involved directly in the crc calculation and only affect the gamestate with certain unit/building types.

My guess is that these causes are mainly responsible for the mismatches that happen out there.

[Fighter19]

That's true! There are quite a lot of cases, where code is being UB. There are a lot of uninitialized member pointers, that could happen to just point to valid values and to random stuff.

I haven't fully understood GameMemory.cpp and to be honest I think whatever it does, is not necessary anymore with current sanitizers. (As far as I understand it's mostly there to track where allocations happened and to put them into pools, something that compilers now do themselves, as far as I know. And even if not, when I tested it on an unrelated project, no real speed gain was observed from pooling objects as opposed to just calling new).
Even OGRE3D as well as glib abandoned their MemorySliceAllocator in favor of simple "new"s, if I'm not mistaken.

That's why I have written a stub for GameMemory.cpp, which allows sanitizers to run and thus stops hiding a bit of UB.

However the memory allocator should also run deterministically, so the undefined behavior should replicate the same.
(Not to say it shouldn't be fixed, but fixing it can lead to desyncs as well, unfortunately. That's why I'm not a fan of carrying along the 1.04 burden. But I'm also a casual vs CPU player, so my opinion on that doesn't matter too much)

[xezon]

Can you help us get asan into our branch? I have great difficulties getting it to run for win32. Conflict with mss32.dll, conflict with FTH, conflict with 4 byte aligned std::string. It's absurd how much roadblocks there are.

[helmutbuhler]

I'm not sure I can be of much help there, I don't have any experience with asan, but I can look into it. But realistically not before Thursday

[augustresende]

From kris.morness (Discord)

"So I do have some general advice about desync fixing:

• It was during Zero Hour that we lost all our network programmers as a result of attrition from crunch and the studio relocation
• It was totally binary. Using deepCRC was slow as shit and essentially triggered a binary save that is tough to parse.
  Super important to do crc checks every frame. In retail it does this every 100 frames if I recall correctly. So when the game desyncs, the cause is almost certainly impossible to figure out.
• During BFME1 I inherited this system as I became responsible for fixing desyncs on that game.
• The approach I took was to convert the save to be human readable, and instead of writing to disk, I'd write to a memory file instead (100x faster), and then I'd keep a buffer of say the last 30 frames. When a desync occurred, it would write the file out to disk so you would have your history."

[nhatter]

Kris is such a nice guy and so friendly and helpful. Hope we can get him a King of Community Outpost badge for Discord! Our group or Generals wouldn't exist without him :)

[augustresende]

• Related: Implement debug to detect game state mismatch #289

[augustresende]

• Related: 100% mismatch rate when playing Zero Hour online using Silicon-based Mac (with Windows 11 VM) #397

[augustresende]

• Related: Playing sounds exclusively for "Local Player" can mismatch the game  #386