Add custom rules to make DisplayName from testname

[jvcppdev]

googletest impose severe constraints in the test naming and parametrization naming. Typically, people use a more or less elaborated naming convention to map the tests hierarchy into the flat testname namespace.

I propose to add a list of regular expressions in the settings file for translating from googletest names to FullyQualifiedNames (with dots). In this way, the test explorer could properly group tests by namespace, class, etc.

(I have been using a similar approach by creating a custom xslt to import the googletest xml output into jenkins, with nice results)

[csoltenborn]

Sounds like a reasonable enhancement, although you probably mean the tests' DisplayName (the FullyQualifiedName serves as a unique identifier to each test, together with the Source (i.e., the executable containing the test). I have renamed the feature accordingly...

Note that I do not have a lot of time to spend on GTA at the moment, so this is not going to happen in the near future if not contributed by the community (of course, I would be willing to help).

[frboyer]

@csoltenborn, after some tests I did, actually it is the FullyQualifiedName that have to be modified so that Test Explorer will correctly group tests in the "Test Hierarchy" and in group by "Namespace" or "Class" modes. But it is the FullyQualifiedName in VsTestCase, not the one in TestCase. I compiled a version of TAfGT with modified ToVsTestCase and ToTestCase (to reverse the modifications) and have been able to correctly display tests in groups. Test Explorer uses the two last dots of the FullyQualifiedName to split the categories.

If I send you some pull requests on bug fixes, or for this extension, do you have time to review them or should I try to send them to MS's TAfGT? And do you know if MS plans to merge back work done on this repo?

[csoltenborn]

It's a good question whether MS will pull changes into the TAfGT repository - I'm currently trying to figure this out...

Anyways: thanks already for looking into that stuff! Howver, I'm not sure that I have understood what you want to achieve: The test hierarchy view seems to working fine (see creenshot below), it shows test executable, fixture, and test name hierarchies. CPP files and namespaces appear to be missing, though (but the latter is probably difficult to achieve due to anon namespaces)...

Would you mind to post a screenshot which shows "your" hierarchy?

Having said that, I would be more than happy to review and pull in your pull request! Let's just first agree on what's supposed to happen...

[frboyer]

This is a possible hierarchy I would like:

The 8 "TypeParametrizedTests" are grouped together, instead of 4 with "/0" and 4 with "/1" (I usually do not know what type "/0" and "/1" are referring to, so I think they are useless to display). And types for the test are grouped together under the test name. (Note that this is an actual screenshot with a modified test adapter, but "run selected tests" is not yet working in that version.)

Of course, I would like to have a 5th hierarchy level, to group the "Arr", but I have not been able to do that. It seems to be a limitation of "Test Explorer" (when looking at tests in C#, it is always 4 levels: "project | namespace | class | method" ; the namespace can contain dots but it will not produce deeper hierarchy). For this reason, some people might like to have "Arr | TypeParametrizedTests | CanIterate<std::array<int,3> >" instead of the hierarchy in my screenshot. Thus, it would be good to have this parametrized.

What do you think?

[csoltenborn]

I see... Are you especially after parameterized or type parameterized tests? If this is the case, we should maybe also consider #102 (which to my understanding is the VS notion of parameterized tests)...

[csoltenborn]

While looking after #102, I found Guidelines for FullyQualifiedNames...

[frboyer]

After doing several tests and looking at the source code of xUnit and data driven tests, I have been able to get the 5th hierarchy level:

The namespace is considered to be "Arr" and the class "TypeParametrizedTests". Also, when displayed in some "group by" mode, the different types are still grouped under the method name.
To have this, there is one thing in the guidelines for FullyQualifiedNames that we must not follow : tests must have the same FullyQualifiedName. More precisely, from experimentation, I got that the tests to be grouped together must have:

• Same FullyQualifiedName (i.e. without the parameters);
• Different Id (note that default Id is based on Source, ExecutorUri and FullyQualifiedName, which are the same in this case, thus we need to generate a different Id);
• Different DisplayName, containing the parameters, but with prefix equal to the FullyQualifiedName;
• Dots to separate namespace (can contain dots), class, and method names.

For example, in the screenshot, both CanDefeatMath tests have FullyQualifiedName = "Arr.TypeParameterizedTests.CanDefeatMath", and the first one has DisplayName = "Arr.TypeParameterizedTests.CanDefeatMath.<MyStrangeArray>". I used a dot before the template type to not have the method name also in the 5th level, with the type.

[csoltenborn]

Nice job! I have also seen your comment on #102, and I had the same line of thought: With data driven tests, one can only execute a certain test after the tests have been executed at least once (before, the data values are not known to the test framework). I think that one can provide the data values through property bags such that they are displayed in the test explorer (no idea how, though), but the problem above remains, and figuring out the test(s) to be run from that information might be more difficult than (and is at least different to (and thus has to be implemented)) using "normal" tests.

Thus, it currently feels like your approach is even preferrable to data-driven tests, because it provides the same benefits, but no drawbacks afaics. However, that probably only holds from VS2017 on, since the hierarchy view has not been available before that version.

I will in the course of today file a question at the VsTest repository and ask whether there are advantages/drawbacks we haven't thought of - maybe we can get some advise on wich way to go... Link to question will follow.

Out of interest: can you post a screenshot of your notion of test names without hierarchy view enabled? Just would like to see how they look like...

As a side note, if you already have some code you would like to share, feel free to create a pull request - you will then get CI for free...

[csoltenborn]

I have create the issue linked above - feel free to add your questions in case I have forgotten anything!

[frboyer]

As you asked, here are some screenshots without hierarchy view, in both VS2015 and VS2017.
In VS2015, there is no "group by namespace" view, so I used the "group by class" to see the display.
VS2015 with standard GTA:

VS2015 with modified GTA:

The types are together in the same "class", compared to standard GTA where there was a /0 and /1. The other "TypeParameterizedTests" we see in this view is for "Vec.TypeParameterizedTests". The types are not in another hierarchy level like they are under VS2017.
VS2017 with modified GTA (same "group by class" view as above):

Note: When I compile your version of GTA, there are 10 tests that fail in GoogleTestAdapter.VsPackage, because adapterTypeName="Microsoft.VisualStudio.TestTools.TestTypes.Unit.UnitTestAdapter" instead of adapterTypeName="executor://googletestrunner/v1". In the golden files from the version in MS repo, the adapterTypeName matches with what my compilation gives (thus tests pass if I copy the golden files from that repo). Am I compiling something wrong?

[frboyer]

Ok, I found the problem with the tests. I had to set environment variable GTA_TESTS_VS2017 because GetVsTestConsolePath was not returning the path to the correct version of VS. Now all tests are passing.
Probably that GTA should use vswhere to get the path, but for now I edited the Development Knowledge Base about that variable.