Finding offsets of fields within structs in a generalized way

[Korporal]

I'm finding it hard going to get a clear and definite answer to this question:

Does the Marshal.OffsetOf when applied to structs that satisfy the unmanaged constraint, always yield the same result as computing an offset by using unsafe code to subtract a structure's address from the field's address?

The documentation for Marshal.OffsetOf doesn't state this if it is the case, it only says this:

OffsetOf provides the offset in terms of the unmanaged structure layout, which does not necessarily correspond to the offset of the managed structure layout.

If the pointer subtraction is always guaranteed to be identical to the Marshal.OffsetOf for a purely unmanaged struct, then do the team agree that this should ideally be stated clearly in the documentation? If these are not guaranteed to be the same, then what options are there for getting this offsetof in a way that's always correct for any target/platform/architecture?

If some context is helpful here, we are using unmanaged structs with pointers to refer to unmanaged memory (e.g. a memory mapped region in the processes address space) and the code needs to be confident that the offset of a field is correct and does indeed refer to the fields actual offset at runtime. I know that field ordering is guaranteed to be the same as the lexical ordering that appears in the source code (if the StructLayout is Sequential) but so far as offsets are concerned we're assuming stuff that might (always) not be true.

[tannergooding]

Marshal.* in general applies to the marshalled (and therefore unmanaged) layout, not the managed layout.

If the type is blittable, then the results should be the same. -- Noting that unless you set DisableRuntimeMarshallingAttribute, some types like bool are not blittable despite being where T : unmanaged

Marshal.* APIs also tend to be very expensive and not optimized or constant folded by the JIT. Many things like AllocHGlobal or SizeOf have effectively been replaced by alternatives (like Unsafe.SizeOf/sizeof() for unmanaged types or NativeMemory.Alloc).

A concept like OffsetOf is very difficult to represent as a managed API. There are proposals like #94976, #94975, and #93946 which covers the ask to expose something that would work for this, but the actual logic to get the offset, especially via a simple helper API, isn't itself complex to write and use internally for an app.

[tannergooding]

If you're having to reflect to get access to the fields, then you're doing unsafe things you likely shouldn't be doing in the first place as its relying on internal implementation details of the type.

You should only be getting field offsets for types that you own and where the code can directly access the field in question.

[Korporal]

Well this was intended for user/customer defined structs if that's what you mean by "own", it was never intended for types defined within the framework or anything, if that's what you mean.

[tannergooding]

It means types that you have written in your own libraries and where you directly have access to the relevant fields using normal C# code.

If you have to use anything like reflection to access the fields, due to them being private or internal for example, then you are doing unsafe things and shouldn't be relying on the type layout (and therefore shouldn't need offsetof)

[Korporal]

Well I'm not concerned with access to private data and so on, although the reflection API supports that so I don't see the significance. All I want is a reliable way to get the offset within a struct, of any field (public but not important here). There are use cases for this but no official support yet.

[tannergooding]

although the reflection API supports that so I don't see the significance

Having the capability to do something doesn't necessarily mean that it's supported. The docs make it very clear that accessing private members is dangerous, could fail arbitrarily, etc.

All I want is a reliable way to get the offset within a struct, of any field (public but not important here). There are use cases for this but no official support yet.

Right, it's an overall very niche thing and with a very trivial implementation for a user to write. While the expense for the language or runtime to implement it is quite high (especially for the language).

[Korporal]

OK I found that code, written in 2010, this worked fine but was based on reverse engineering and lots of hacking so the comments are likely inaccurate at best and of course pertain to a much older runtime. This is core of the logic, there's more (we first see if the data is already in a static cache etc) but you get the idea.

This is generic code and likely still works or could be updated to, but this is far from straightforward which is why a proper official implementation written by people with a deeper understanding, would be so useful.

            // Now search for the field with the specified Name. When we find this, we must construct a dynamic
            // IL function to calculate the fields offset. This is done by creating an instance of the struct
            // and then subtracting the invoker of the field from the invoker of the struct itself.
            // There is no managed support for doing this or 3rd party libraries.

            foreach (FieldInfo field in fields)
            {
                if (field.Name == Name)
                {
                    dynamite = new DynamicMethod("get_field_offset", typeof(long), dynargs, typeof(Runtime), true);
                    generator = dynamite.GetILGenerator();

                    /*----------------------------------------------------------------------*/
                    /* Create a local instance of the Type, we do this so that we can       */
                    /* then take its invoker. It seems that declaring a local does not      */
                    /* run or require any constructor. Thus we can always execute this      */
                    /* declare operation even for classes that have a private parameterless */
                    /* constructor.                                                         */
                    /*----------------------------------------------------------------------*/

                    if (Type.IsValueType)
                       {
                       local_0 = generator.DeclareLocal(Type);
                       local_1 = generator.DeclareLocal(typeof(long));
                       local_2 = generator.DeclareLocal(typeof(long));
                            
                       generator.Emit(OpCodes.Nop);
                       generator.Emit(OpCodes.Ldloca_S, local_0);
                       generator.Emit(OpCodes.Ldflda, field);
                       generator.Emit(OpCodes.Conv_U);
                       generator.Emit(OpCodes.Conv_U8);
                       generator.Emit(OpCodes.Stloc_1);
                       generator.Emit(OpCodes.Ldloca_S, local_0);
                       generator.Emit(OpCodes.Conv_U);
                       generator.Emit(OpCodes.Conv_U8);
                       generator.Emit(OpCodes.Stloc_2);
                       generator.Emit(OpCodes.Ldloc_1);
                       generator.Emit(OpCodes.Ldloc_2);
                       generator.Emit(OpCodes.Sub);
                       generator.Emit(OpCodes.Ret);
                       }
                    else
                       {
                       /* THIS CODE APPEARS TO WORK BUT REQUIRES CLASS TO HAVE A DEFAULT NO-ARGS CONSTRUCTOR */
                       /* WE NEED TWO VERSIONS OF THIS, ONE THAT INSTANTIATES AN OBJECT THE OTHER THAT       */
                       /* JUST ACCEPTS AN ALREADY EXISTING OBJECT (OR RATHER 'REF' TO ONE) THIS LATTER FORM  */
                       /* MAY BE USEFUL WHEN WE DON'T HAVE A CONSTRUCTOR...                                  */
                       
                       local_0 = generator.DeclareLocal(typeof(IntPtr));
                       local_1 = generator.DeclareLocal(typeof(ulong));
                       local_2 = generator.DeclareLocal(typeof(ulong));

                       generator.Emit(OpCodes.Nop);
                       generator.Emit(OpCodes.Ldloca,0);
                       generator.Emit(OpCodes.Newobj,cons[constructor_to_use]);
                       generator.Emit(OpCodes.Stind_Ref,local_0);
                       generator.Emit(OpCodes.Ldloca,0);
                       generator.Emit(OpCodes.Ldind_Ref,local_0);
                       generator.Emit(OpCodes.Ldflda, field);
                       generator.Emit(OpCodes.Conv_U);
                       generator.Emit(OpCodes.Conv_U8);
                       generator.Emit(OpCodes.Ldloc_0);
                       generator.Emit(OpCodes.Conv_U);
                       generator.Emit(OpCodes.Conv_U8);
                       generator.Emit(OpCodes.Sub);       // push A, push B, eval A - B
                       generator.Emit(OpCodes.Ret);
                       
                       }

                    try
                    {
                    function = (GetOffset)(dynamite.CreateDelegate(typeof(GetOffset)));
                    ofx = (int)function();
                    }
                    
                    catch (Exception e)
                    {
                    ofx = -1;
                    }
                    
                    finally
                    {
                    abs_field_offset_cache[Type].Add(Name, new TypeOffset(field.FieldType, ofx));
                    }   
     
                    return (ofx);

[Korporal]

Well that code worked, needed a tiny bit of rejigging but it works and is generic and doesn't care if the struct is unmanaged even. But surely this could be encapsulated and made part of the runtime?

[Korporal]

Here's a little static class I created, the code builds and runs and lets you get the offset of any field in any struct (or class, but that's not something I'm concerned about here).

    public static class Structures
    {
        private delegate long GetOffset();
        private static Dictionary<Type, Dictionary<string, TypeOffset>> abs_field_offset_cache = new Dictionary<Type, Dictionary<string, TypeOffset>>();
        private class TypeOffset
        {
            private Type type;
            private int offset;

            public Type Type
            {
                get { return type; }
            }

            public int Offset
            {
                get { return offset; }
            }

            public TypeOffset(Type Type, int Offset)
            {
                type = Type;
                offset = Offset;
            }
        }

        public static int OffsetOf(Type Type, String Name)
        {
            Type[] dynargs = { };
            DynamicMethod dynamite;
            ILGenerator generator;
            LocalBuilder local_0, local_1, local_2;
            GetOffset function;
            bool has_empty_constructor = false;
            int constructor_to_use = 0;
            ConstructorInfo[] cons = null;
            int ofx = 0;

            if (Type.IsValueType == false)
            {
                cons = Type.GetConstructors(BindingFlags.Instance | BindingFlags.Public | BindingFlags.NonPublic);

                if (cons.Length == 0)
                    has_empty_constructor = true;
                else
                    for (int I = 0; I < cons.Length; I++)
                    {
                        if (cons[I].GetParameters().Length == 0)
                        {
                            has_empty_constructor = true;
                            constructor_to_use = I;
                        }
                    }

                if (has_empty_constructor == false)
                    return (-1);

            }

            //lock (abs_field_offset_cache)  // HWG005
            try
            {
                //GlobalNoSyncLock.Lock(GlobalLockType.AbsFieldOffsetCache);
                if (abs_field_offset_cache.ContainsKey(Type))
                {
                    if (abs_field_offset_cache[Type].ContainsKey(Name))
                        return (abs_field_offset_cache[Type][Name].Offset);

                }
                else
                {
                    // Add the Type's GUID and an new string/int dictionary, to the cache.
                    abs_field_offset_cache.Add(Type, new Dictionary<string, TypeOffset>());
                }

                //if (Runtime.ContainsOnlyValueTypes(Type) == false)
                //   throw new ArgumentException("The Type must be a pure value Type in order to determine a field's managed offset.");

                // Get every field declared in the struct.

                FieldInfo[] fields = Type.GetFieldsInDeclaredOrder(BindingFlags.Instance | /*BindingFlags.DeclaredOnly*/ BindingFlags.Public | BindingFlags.NonPublic);

                // Now search for the field with the specified Name. When we find this, we must construct a dynamic
                // IL function to calculate the fields offset. This is done by creating an instance of the struct
                // and then subtracting the address of the struct from the address of the field.
                // There is no managed support for doing this or 3rd party libraries.

                foreach (FieldInfo field in fields)
                {
                    if (field.Name == Name)
                    {
                        dynamite = new DynamicMethod("get_field_offset", typeof(long), dynargs, typeof(Utility), true);
                        generator = dynamite.GetILGenerator();

                        /*----------------------------------------------------------------------*/
                        /* Create a local instance of the Type, we do this so that we can       */
                        /* then take its invoker. It seems that declaring a local does not      */
                        /* run or require any constructor. Thus we can always execute this      */
                        /* declare operation even for classes that have a private parameterless */
                        /* constructor.                                                         */
                        /*----------------------------------------------------------------------*/

                        if (Type.IsValueType)
                        {
                            local_0 = generator.DeclareLocal(Type);
                            local_1 = generator.DeclareLocal(typeof(long));
                            local_2 = generator.DeclareLocal(typeof(long));

                            generator.Emit(OpCodes.Nop);
                            generator.Emit(OpCodes.Ldloca_S, local_0);
                            generator.Emit(OpCodes.Ldflda, field);
                            generator.Emit(OpCodes.Conv_U);
                            generator.Emit(OpCodes.Conv_U8);
                            generator.Emit(OpCodes.Stloc_1);
                            generator.Emit(OpCodes.Ldloca_S, local_0);
                            generator.Emit(OpCodes.Conv_U);
                            generator.Emit(OpCodes.Conv_U8);
                            generator.Emit(OpCodes.Stloc_2);
                            generator.Emit(OpCodes.Ldloc_1);
                            generator.Emit(OpCodes.Ldloc_2);
                            generator.Emit(OpCodes.Sub);
                            generator.Emit(OpCodes.Ret);
                        }
                        else
                        {
                            /* THIS CODE APPEARS TO WORK BUT REQUIRES CLASS TO HAVE A DEFAULT NO-ARGS CONSTRUCTOR */
                            /* WE NEED TWO VERSIONS OF THIS, ONE THAT INSTANTIATES AN OBJECT THE OTHER THAT       */
                            /* JUST ACCEPTS AN ALREADY EXISTING OBJECT (OR RATHER 'REF' TO ONE) THIS LATTER FORM  */
                            /* MAY BE USEFUL WHEN WE DON'T HAVE A CONSTRUCTOR...                                  */

                            local_0 = generator.DeclareLocal(typeof(IntPtr));
                            local_1 = generator.DeclareLocal(typeof(ulong));
                            local_2 = generator.DeclareLocal(typeof(ulong));

                            generator.Emit(OpCodes.Nop);
                            generator.Emit(OpCodes.Ldloca, 0);
                            generator.Emit(OpCodes.Newobj, cons[constructor_to_use]);
                            generator.Emit(OpCodes.Stind_Ref, local_0);
                            generator.Emit(OpCodes.Ldloca, 0);
                            generator.Emit(OpCodes.Ldind_Ref, local_0);
                            generator.Emit(OpCodes.Ldflda, field);
                            generator.Emit(OpCodes.Conv_U);
                            generator.Emit(OpCodes.Conv_U8);
                            generator.Emit(OpCodes.Ldloc_0);
                            generator.Emit(OpCodes.Conv_U);
                            generator.Emit(OpCodes.Conv_U8);
                            generator.Emit(OpCodes.Sub);       // push A, push B, eval A - B
                            generator.Emit(OpCodes.Ret);

                        }

                        try
                        {
                            function = (GetOffset)(dynamite.CreateDelegate(typeof(GetOffset)));
                            ofx = (int)function();
                        }

                        catch (Exception e)
                        {
                            ofx = -1;
                        }

                        finally
                        {
                            abs_field_offset_cache[Type].Add(Name, new TypeOffset(field.FieldType, ofx));
                        }

                        return (ofx);
                    }

                }
            }

            catch (Exception E)
            {
                ;
            }

            finally
            {
                ;// GlobalNoSyncLock.Unlock(GlobalLockType.AbsFieldOffsetCache);
            }

            throw new ArgumentException("The specified field Name does not exist within the Type: '" + Type.Name + "'.");
        }
        public static FieldInfo[] GetFieldsInDeclaredOrder(this Type Type, BindingFlags Flags)
        {
            return (Type.GetFields(Flags).OrderBy(f => f.MetadataToken).ToArray());
        }
    }

[Korporal]

This is a use case, I grabbed this simple example from this Github code:

typedef struct mya_header {
    /*
     * Holds the user data size in bytes and status flags of
     * the previous adjacent block.
     */
    size_t prev_info;

    /*
     * Holds the user data size in bytes and status flags of
     * the current block.
     */
    size_t curr_info;

    /*
     * If the current block is not in use, holds a pointer to
     * the previous block in the free list.
     */
    struct mya_header *prev_free;

    /*
     * If the current block is not in use, holds a pointer to
     * the next block in the free list.
     */
    struct mya_header *next_free;
} mya_header_t;

This prefixes blocks of memory created by an allocator (heap). As that examples points out the data space used by the two freelist pointers is not wasted, it is part of the user space granted to the caller, the address they see when a block is allocated is in fact the address of that prev_free pointer but they just overwrite it with their own data, only when it gets freed do we use the bytes in those two pointers as pointers.