AccessViolationException when serializing two worlds, but deserializing in different order

[solonrice]

Thank you for implementing the serialization! I recently had a chance to try it out and found an issue when using multiple worlds. The TLDR is basically this:

If you create two worlds and add an entity to each (with a component), serialize both worlds, then deserialize them later in a different order and in a different process, an AccessViolationException is thrown in the MessagePack formatters.

It looks like the exception is thrown almost every time in the ChunkFormatter implementation when the chunk array is setup for the first component type.

Here is some code to help reproduce it:

    [Fact]
    public void TestSerialized()
    {
        // Prove compression works
        Assert.Equal("Compression is working", Encoding.UTF8.GetString(Decompress(Compress(Encoding.UTF8.GetBytes("Compression is working")))));
        
        // Set up binary serializer with no extra formatters
        var binarySerializer = new ArchBinarySerializer();
        
        // Make two worlds, adding entities with different components to each (doesnt seem to matter which two types, just that they are different).
        var worldA = World.Create();
        worldA.Create(12);
        var worldB = World.Create();
        worldB.Create(13f);
        
        // Serialize both worlds in order (A, B).
        var serializedA = Convert.ToBase64String(Compress(binarySerializer.Serialize(worldA)));
        var serializedB = Convert.ToBase64String(Compress(binarySerializer.Serialize(worldB)));

        // Output the compressed strings for copy/paste into deserialize method.
        _testOutputHelper.WriteLine(serializedA);
        _testOutputHelper.WriteLine("-=-=-=-=-");
        _testOutputHelper.WriteLine(serializedB);
    }

    [Fact]
    public void TestDeserialized()
    {
        // Set up a binary serializer again
        var binarySerializer = new ArchBinarySerializer();
        
        // Paste in the two compressed serialized world strings
        var serializedA = "7dlBaoNAFIDhF4jQRQ7hAURCi5gsssqqtItASvdqH0ViFMax4DH0YFlkV/AuVm16hS7i//z5cEAYmJ1M0z3IomcYhmEYZm7TdktpRdrhjYiIiIiIZtD4C8A5EBERERHNpouILJrJ789jXVo9+8+5fXr03NvqYNKvyKq/L4y+prHnvqsp0yLfhf56fDx3X2W2MrrLtbImyjz3UMVZmrxo/VacdPgw0WQTfISxRmG43ei427LtZdr1DyfonOA6LAQAAAAAAAAAAAAAAADuj/+8i/q9dlpx7AAAAAAAAAAAAAAAAPfHDw==";
        var serializedB = "7dlBasJAGIbh32Kgx8gBQnATYhcuiktdBBT3SfrTDo0JjIngMYy3cuHCXSF3STOlvUIX5v18eXBAGMgueO6eZdIzxhhjbGxru6m0Iu3wjYiIiIiIRpB7BeA5EBERERGNppuITM7Op6+PzelQ6z7cmPK90MD/PSbWHNNaw2VldW2ywN+pPZiqXMThzH0Cf9kUdWN1UWpT27QI/KTJCpOv9LStPnX4Ya75PHqLM03j+GWu7rrppe/l5/Y/vKjzovtwEAAAAAAAAAAAAAAAAHg8/vXfKC+6viYiV3cxAAAAAAAAAAAAAAAAPBTf";

        // Deserialize in different order (B, A). Will throw on first one.
        var worldB = binarySerializer.Deserialize(Decompress(Convert.FromBase64String(serializedB)));
        Assert.True(worldB.Size > 0);
        
        var worldA = binarySerializer.Deserialize(Decompress(Convert.FromBase64String(serializedA)));
        Assert.True(worldA.Size > 0);
    }
    
    private static byte[] Compress(byte[] data)
    {
        using var output = new MemoryStream();
        using (var dstream = new DeflateStream(output, CompressionLevel.SmallestSize)) {
            dstream.Write(data, 0, data.Length);
        }
        return output.ToArray();
    }
    
    private static byte[] Decompress(byte[] data)
    {
        using var input = new MemoryStream(data);
        using var output = new MemoryStream();
        using (var dstream = new DeflateStream(input, CompressionMode.Decompress)) {
            dstream.CopyTo(output);
        }
        return output.ToArray();
    }

You can simply run the TestDeserialized() method, which has the encoded world already there. But of course, prove it out completely by running TestSerialized(), copying the output of each world (A and B) to the TestDeserialized() method, then running that method to show the AccessViolationException.

Thanks again for an awesome ECS!

EDIT: This also happens with the ArchJsonSerializer, for what it's worth.

[genaray]

Thank you!
I looked at the whole thing earlier. Quasi just dragged it into Arch.Extended.Sample. Because I updated Arch.Persistence in the meantime because of the new arch release, I had to use new compressed strings.

Anyway, I could run it without problems and there were no errors :/ Where did you include the whole thing? Is there some more information?

[solonrice]

You know, it just occurred to me that I had the Arch.Extended repo checked out and added directly to my solution. I’ll try again with nuget package and see if different result.

[genaray]

You know, it just occurred to me that I had the Arch.Extended repo checked out and added directly to my solution. I’ll try again with nuget package and see if different result.

I had it also checked out and well... it worked for me :o
Since another user has a similar issue... i actually believe its some sort of cpu/os/environment thing. What os are ya using? And what engine? Could you probably debug the iusse yourself to see why it fucks up and why its happening?

[dacete]

I'm also getting this error

Fatal error. System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
   at Arch.Core.Chunk.GetArray(Arch.Core.Utils.ComponentType)
   at Arch.Persistence.ChunkFormatter.Deserialize(MessagePack.MessagePackReader ByRef, MessagePack.MessagePackSerializerOptions)
   at Arch.Persistence.ArchetypeFormatter.Deserialize(MessagePack.MessagePackReader ByRef, MessagePack.MessagePackSerializerOptions)
   at Arch.Persistence.WorldFormatter.Deserialize(MessagePack.MessagePackReader ByRef, MessagePack.MessagePackSerializerOptions)
   at MessagePack.MessagePackSerializer.Deserialize[[System.__Canon, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](MessagePack.MessagePackReader ByRef, MessagePack.MessagePackSerializerOptions)
   at MessagePack.MessagePackSerializer.Deserialize[[System.__Canon, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](System.ReadOnlyMemory`1<Byte>, MessagePack.MessagePackSerializerOptions, System.Threading.CancellationToken)
   at Arch.Persistence.ArchBinarySerializer.Deserialize(Byte[])
   at Game.MainMenu.OnAwake()
   at FlaxEngine.Interop.NativeInterop+Invoker+InvokerNoRet0`1[[System.__Canon, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].InvokeThunk(System.Object, FlaxEngine.Interop.ManagedHandle, IntPtr*)
   at FlaxEngine.Interop.NativeInterop+ThunkContext.InvokeThunk(FlaxEngine.Interop.ManagedHandle, IntPtr, IntPtr, IntPtr, IntPtr, IntPtr, IntPtr, IntPtr)
Segmentation fault

It only happens in FlaxEngine builds, not in editor. And this is the code that I use to cause it:

        var world = World.Create();
        world.Create(new TestStruct3());
        var serializer = new ArchBinarySerializer();
        var bytes = serializer.Serialize(world);
        world = serializer.Deserialize(bytes);

edit: where TestStruct3 just holds an int

[dacete]

I think problem comes from this lookup array and the componentType ID. I am gonna go deeper to see why the lookup array is wrong or if the ID itself is wrong, but maybe you got a clue from this

[genaray]

Thanks for investigating! Excited what you might else discover ^^
Could it be related to your other issue genaray/Arch.Extended#40 (comment) ?

[dacete]

For some reason, when you convert sourceArray.GetType().GetElementType() to (ComponentType), eventually it tries to do ComponentRegistry.TypeToComponentType.TryGetValue, the correct type is inside the dict as key, but it somehow fails the TryGet and thus it Adds it to the component registry once more, generating a duplicated entry

[genaray]

For some reason, when you convert sourceArray.GetType().GetElementType() to (ComponentType), eventually it tries to do ComponentRegistry.TypeToComponentType.TryGetValue, the correct type is inside the dict as key, but it somehow fails the TryGet and thus it Adds it to the component registry once more, generating a duplicated entry

Well that explains the issue. However i dont get why it should do this... This would mean that the type is "different" somehow.

[genaray]

Is this still the case in the more recent versions of arch and arch extended?

[dacete]

I'm using recent versions of Arch and Extended serialization, I've tested only binary so far but I didn't run into this anymore.