TestAdaptor does not run ModuleInitializer before enumerating DefinedTypes

[wasabii]

I have a situation where I have a test assembly which depends on another assembly. However, the location of the second assembly needs to be determined at runtime, because it may be environmentally variable. For instance, there may be a different version of that assembly when running libraries under different platforms.

In order to accomplish this, the first assembly (the assembly with tests) uses a ModuleInitializer (.cctor) to hook into AppDomain.AssemblyResolve and customize resolution of it's dependent assembly that may vary by location.

This works properly in normal execution scenarios. If you run code in any of the methods, the module's cctor gets executed, the resolve hook is added, and then the calls succeed.

However, MSTest does not run the module initializer when enumerating and searching for test methods. It does this purely through reflection. But a consequence of that is that reflection causes a load of the types being reflected: including their base classes. Some of which might be from the dependent assembly. Thus causing an initialization of it before the module cctor might run.

Though I do think this is somewhat of a unique circumstance, I also think that it would make sense that MSTest (and other testing frameworks) that load the assembly in the non-reflection-only context, with the intent to scan and load all of the types, should ensure the module cctor runs before scanning too deep into the types of the assembly. This can be done by calling RuntimeHelpers.RunModuleConstructor immediately after loading the assembly, and before enumerating it's types.

[nohwnd]

Do you see any downsides to doing this? We would be at least changing behavior for the existing tests if they already implement module initializer, and depend on it not being run in tests. Kinda using the opposite effect, where they "conditionally" setup something when not running under test.

[Evangelink]

I need to make a test, I am wondering if this would not work out of the box with the runner as we are then a simple executable so default logic should be running. If that's the case, I'd prefer to rely to suggest people to move to the runner rather than adding some extra logic.

[wasabii]

I hadn't even heard of the runnner.

Does the runner execute code in the assembly before it reflects over it?

[nohwnd]

Yes, it builds an exe and runs that exe to run tests. The integration to VS is not full right now, but in CI it could work. This is the docs: https://learn.microsoft.com/en-us/dotnet/core/testing/unit-testing-mstest-runner-intro

[wasabii]

I'll give it a try. If it works as advertised, I'll close this bug in agreement. No need to modify the adapter if an alternative path exists.

I guess it injects a main method or something? That takes arguments to conduct discovery/execution? I have no idea. But that seems like a thing I'd imagine it would have to do.

[nohwnd]

Yes, Program.cs with Main is injected. This is similar to what Microsoft.NET.Test.SDK does. But there is one fundamental difference.

Microsoft.NET.Test.SDK (which is coming from TestPlatform aka vstest), the Main method is inserted to force the test project to generate deps.json and runtimeconfig.json, and then the dll is loaded (via reflection) into testhost.exe. Circumventing some of the built-in setups that .net runtime might do.

In mstest runner, we inject the Main method, and then run the exe, fully relying on .net runtime to do all the built-in setups. This is why Evangelink thinks that this will simply work without any change.

[wasabii]

I'm going through the code. Where is the main method injected? I don't see it happening in any of the build scripts....

[wasabii]

Oh. I think I found it. It's a SourceGenerator. Bummer. Won't work for me. My project isn't C#. So I guess my user's would have to write their own Main method.

[nohwnd]

It is not source generator, it is done via msbuild. All C#, F# and VB should be supported. Look for Name="_GenerateTestingPlatformEntryPoint" target.

[Evangelink]

Please find enclosed a little demo project showing that the ModuleInitializer is correctly handled and called with MSTest runner.
DemoModuleInitializer.zip

We strongly recommend you to update your projects to rely on this instead of VSTest platform. On the link shared by @nohwnd, you will find more info about integration with VS, CI, and dotnet cli.

I'll close the issue on MSTest sidebut if you have a strong need for it to be supported in VSTest, please open an issue there