Test discovery appears to run the whole test suite

[petebannister]

Not sure if I have something configured wrong here. Test discovery can be a bit slow. I was actually trying to debug another issue which looks a lot like #328 but noticed something..

I've put in some OutputDebugString into by test application to see what args it gets called with and this is the log seen in sysinternals DebugView:

[5896] Informational: ========== Starting test discovery ========== 
[52048] DBGHELP: Symbol Search Path: . 
[52048] DBGHELP: C:\work\MyTest\_build\VC\Win32\Debug\MyTest.exe - OK 
[52048] DBGHELP: .\MyTest.pdb - file not found 
[52048] DBGHELP: .\symbols\exe\MyTest.pdb - file not found 
[52048] DBGHELP: MyTest - private symbols & lines  
[52048]         C:\work\MyTest\_build\VC\Win32\Debug\MyTest.pdb 
[5896] Warning: Could not locate debug symbols for 'C:\work\MyTest\_build\VC\Win32\Debug\MyTest.exe'. To make use of '--list_content' discovery, ensure that debug symbols are available or make use of '<ForceListContent>' via a .runsettings file. 
[23756] TEST: C:\work\MyTest\_build\VC\Win32\Debug\MyTest.exe
[5896] Informational: Found 283 tests in executable C:\work\MyTest\_build\VC\Win32\Debug\MyTest.exe 
[5896] Informational: ========== Test discovery finished: 283 Tests found in 21.2 sec ========== 

The line:

[23756] TEST: C:\work\MyTest\_build\VC\Win32\Debug\MyTest.exe

Is showing the args it is called with.

So basically it looks like it runs the full test suite during test discovery & executes the program with no arguments. Is this to be expected?

The output from above is for a test project that is built from a vcxproj in VS2019 16.11.9. .NET framework 4.8.04084.
Google Test Adapter is not included in VS Installer and the extension is used with version 0.18.0.1432.

It does show Google Test Adapter 1.0 in the 'Installed products' list in VS2019 about box which seems odd. So not sure if the MS fork is left in there from a prior installation.

Might be useful if the debug message included the version of the test adapter as I'm not able to tell if it is definitely using the correct one. Looking at a detailed trace from the test discovery does look like it is the extension though:

TpTrace Verbose: 0 : 9892, 3, 2022/01/18, 10:36:48.719, 5701571625163, testhost.x86.exe, TestPluginCache: Discoverers are 'Microsoft.VisualStudio.TestPlatform.Extensions.CodedWebTestAdapter.CodedWebTestDiscoverer, Microsoft.VisualStudio.TestPlatform.Extensions.CodedWebTestAdapter, Version=16.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a,Microsoft.VisualStudio.TestPlatform.Extensions.GenericTestAdapter.GenericTestDiscoverer, Microsoft.VisualStudio.TestPlatform.Extensions.GenericTestAdapter, Version=16.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a,Microsoft.VisualStudio.TestPlatform.Extensions.OrderedTestAdapter.OrderedTestDiscoverer, Microsoft.VisualStudio.TestPlatform.Extensions.OrderedTestAdapter, Version=16.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a,Microsoft.VisualStudio.TestPlatform.Extensions.VSTestIntegration.MSTestDiscoverer, Microsoft.VisualStudio.TestPlatform.Extensions.VSTestIntegration, Version=16.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a,Microsoft.VisualStudio.TestPlatform.Extensions.WebTestAdapter.WebTestDiscoverer, Microsoft.VisualStudio.TestPlatform.Extensions.WebTestAdapter, Version=16.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a,Microsoft.VisualStudio.TestTools.CppUnitTestFramework.CppUnitTestExtension.TestDiscoverer, Microsoft.VisualStudio.TestTools.CppUnitTestFramework.CppUnitTestExtension, Version=16.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a,Microsoft.VisualStudio.Workspace.ExternalBuildFramework.TestProvider.ExternalBuildProjectDiscoverer, Microsoft.VisualStudio.Workspace.ExternalBuildFramework.TestProvider, Version=16.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a,BoostTestAdapter.BoostTestDiscoverer, BoostTestAdapter, Version=1.0.8.10, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a,GoogleTestAdapter.TestAdapter.TestDiscoverer, GoogleTestAdapter.TestAdapter, Version=0.18.0.1432, Culture=neutral, PublicKeyToken=8d7bc984b03f1caf'.

[csoltenborn]

This is not to be expected, and it does not happen ;-)

During discovery, GTA runs the test executable with parameter --gtest_list_tests. If your tests are implemented correctly, this will not run the tests, but just list them to the console. You can try easily by running MyTest.exe --gtest_list_tests on the console. Note that if I remember correctly, it is indeed possible to mess up your tests such that stuff is run when listing tests, but that's not the responsibility of GTA.

Note that if you enable the GTA settings Print test output (and probably choose an appropriate GTA Output Mode - I do not remember completely, try Debug), GTA will print the executed command including parameters to the Test Output window.

You should be able to figure out which adapter(s) you have installed by looking into VS' options - there should (only) be a section Google Test Adapter, but no section Test Adapter for Google Test. Plus, if you enable GTA's option Prefix output with [GTA], this should make also clear which adapter you have running (I think that TAfGT does not have that option).

Finally, note that there will most likely not be a GTA version for VS2022 or later, so it might be a better option to already go for TAfGT (which is a MS clone of this adapter and has many of its features).

Let me know whether this clarifies things - I would then close this issue...

[petebannister]

Perhaps this is just something weird with OutputDebugString API - will investigate further. The test discovery time was suspicious though as just slightly longer than my test suite running time. Perhaps there is something else going on here

[petebannister]

Looks like it is something weird with OutputDebugString.. sorry for false alarm & thank you for getting back to me

[csoltenborn]

Don't worry - thanks for reporting anyways!

Btw, there are some hints in the trouble shooting section on how to speed up test discovery, but they come with a price, as everything in life :-)

How many tests are contained in your executable? You should certainly see the same effect if running your executable with the discovery flag from the console. I would maybe try to debug this - I've seen cases where people accidentally triggered loading of expensive resources already during test discovery (i.e., asa the executable is, well, executed). If you make sure that all such stuff is dealed with at the appropriate gtest setup facilities, they will not affect discovery time at all. But this is sometimes difficulty to see in C++, I find.

[petebannister]

Thanks for this - I think it is the symbol loading that is slow - actually most of the time spent appears to be before it even executes the test application (judging by when outputdebugstring has its moment vs start of test discovery). The PDB of the main executable (which does not contain any tests) is pretty big (580MB) and the test exe PDB is even larger (620MB). I do have a symbol server on VPN connection configured in VS, perhaps that has an effect - hoping to move that to Azure at some point. Perhaps there is some way to reduce the PDB size by moving stuff into DLLs but of course that would adversely affect application load time. Might be good to be able to tell VS not to bother checking the main application for tests in addition to the test executable. It would be handy to have a button on the test view to be able to manually re-run the test discovery process - particularly in the CMake case without having to reconfigure the project.

More of a pressing issue for me is to get googletests built with cmake and added via gtest_discover_tests to have source information in the test adapter. Having trouble finding much info on that at all. It looks like the gtest_discover_tests does not produce anything that contains line/file info (perhaps googletest should really have a --gtest_list_tests mode that includes that information - it could add FILE and LINE to the various TEST macros and avoid scanning the PDB at all!). Looks like gtest_discover_tests actually just parses the source code with a regex to discover the test names anyway so it could concievably provide some info from that process.
It then looks like in Visual Studio it has ExternalBuildFramework ExternalBuildProjectContainerDiscoverer that finds the tests for the test adaptor from the gtest_discover_tests info. That process appears to generate some json files (which must be where the weird EBF.Tests prefix (EBF => ExternalBuildFramework?) on all the tests come from when using cmake). Going by the trace it doesn't look like this even tries to look at the PDB. But the Json files includes a couple of suspiciously interesting keys with no information in them:

"ProjectFile":null,
"LineNumber":-1

So not sure if there is some way to get that info to be populated correctly or not. I tried editing these to add some info in but that does not appear to trick the system.

These json files appear to be used as the test runsettings files.

[petebannister]

Looks like one workaround for that is to generate an sln from cmake. The unit tests are not found but adding the magic my.exe.is_google_test makes it work!

[csoltenborn]

In fact, gtest has an option to generate an XML file containing information on test location (see this issue, but that comes with some drawbacks - I gave it a shot at one time, but the advantages in speed where neglible compared to the effort (at least for what I looked at). Since I'm not actively developing this project any more, this is not going to happen, I fear. If anybody would be willing to take over this project, I could share what I got so far, though...

If the PDBs are received over a (possibly slow) network connection, that would of course explain why test discovery is slow. I guess that this does not happen any more when working with a solution, and thus speeds up the process? If you can confirm this (I do not understand how this is related to cmake since I do not have any experience with that), I would happily add that information to the trouble shooting section (or feel free to open an according PR against the README :-) )...

One things just comes to my mind: It wouldn't be at lot of effort to add an option to ignore certain PDBs (there already is one with which you can add PDBs which otherwise could not be found). But once more, since this adapter is probably not going to support VS2022, it might be a better idea for you to go for TAfGT right now (and to file your feature requests and problems against MS' adapter - although I just realize that they apparently do not support issues, don't know why :-) ).

[petebannister]

Thank you for this - will use the MS one from now on. Tried out this with VS2022 and the results are the same with cmake. Have not really spent much more time looking at the slow loading - was more focused on the issue of not having source info with cmake version... Have still not resolved that but parking it for now as already spent waaay too much time on it!

Thank you for originally creating this extension it has been extremely useful :)