Datasource is not working with MSTest.TestFramework Adapter and MSTest.TestAdapter from version 3.5.0.

[raghere]

We were using MSTest.TestFramework Adapter and MSTest.TestAdapter version 2.2.3 and tried to update it to the latest version 3.6.2. But after updating, [DataSource("TC123456")] does not seem to work. We do not see any error either in the Testlog. It just can not start execution of the test.

We are using app.config file which referes to this Datasource.

We are using Microsoft.NET.Test.Sdk - 17.11.1 and .NET Framework 4.7.2.

Also we noticed that Datasource problems exists with some of the versions.

2.2.5 works
2.2.10 - works

3.0.0 - does not work
3.0.1 - works
3.1.1 - works
3.2.0 - does not work
3.3.0 - works

3.4.0 - does not work
3.4.2 - works
3.4.3 - works

3.5.0 - does not work
3.5.1 - does not work
3.5.2 - does not work
3.6.0 - does not work
3.6.1 - does not work
3.6.2 - does not work

Why are some of the versions giving problem with Datasource ? Could you pleaase help us using the latest version 3.6.2 and still be able to work with Datasource ?

Thanks and Regards.

AB#2300883

[Evangelink]

I just checked and the coverage for datasource is really low which would explain all these changes. Would you mind providing a small complete repo for us to investigate? I have to admit I have limited knowledge on all the ways the data source can be used.

[Youssef1313]

@Evangelink I think I can repro. Let's discuss tomorrow.

[raghere]

@Evangelink Thank you for your quick response. I will provide Repo as soon as I get time for it.

[Youssef1313]

Can you also please check if [assembly: TestDataSourceDiscovery(TestDataSourceDiscoveryOption.DuringExecution)] works around the issue?

[raghere]

@Youssef1313 Hey ! That works. :-)

[Youssef1313]

@Evangelink Can you advise if there are any downsides to using TestDataSourceDiscoveryOption.DuringExecution until this is fixed.

[raghere]

When we use the version which works with Datasource for example 3.4.3 (without the workaround mentioned by @Youssef1313 ), the tests are executed correctly through Yaml Pipeline in Azure Devops, but the Test Results are not shown properly, and in Testexecution in Testplan, the tests remain in status "In Progress" forever.

(We use Microsoft Self hosted Agents which are installed on virtual machines to execute the tests)

The only version where Test results are shown properly is 2.2.3. So we unfortunately can not upgrade MSTest.TestFramework and MSTest.TestAdapter. We would really appreciate your feedback/comments on it, and if you think this is a different issue and should be addressed separately, We would do so.
Thanks.

[Evangelink]

@Evangelink Can you advise if there are any downsides to using TestDataSourceDiscoveryOption.DuringExecution until this is fixed.

Given that the tests are only discovered during execution that means it won't be expanded so it won't be possible to run a single entry of the test. It's also impacting the TRX format a little as the tests are no longer considered as different so you have multiple results for a single test rather than each entry considered as a different test with its own result.