-
Notifications
You must be signed in to change notification settings - Fork 223
Import BuildEnv.props into c# Test projects #459
base: dev
Are you sure you want to change the base?
Conversation
By importing the BuildEnv.props into the C# projects, it is possible to open the solution file directly. i.e. Can load a Unit Test solution directly from Windows File Explorer and then build, Run or Debug the test(s)
Have you tested the results of this with a local build of the test SDK? This is important as the build process will transform project files used in the repository before incorporating them into the MSI. (The build tasks use the term "transmogrify" for this stage - before you ask, I have no idea why...) Thus it is important to ensure that the resulting projects are still usable when installed by the Test kit installer. The conditional nature of the import should allow everything to just work but it is important to check, especially since we don't have anything in the release build process to verify the test kit yet. |
I tested as follows; Delete BuildOutput
Build the sdk with build_sdk.cmd, no parameters. Build my solution with build_solution ds5 5.05 BPPLUSR7 I can then run dotest.cmd mscorlibtest.txt Also run the same tests on my hardware (ARM920T) using RunTest.exe Loaded several, but not all 60+ Test solutions, by opening the solution file and build/deploy/run and in some set breakpoints to check expected behavior. Can either run the tests in the Emulator or against the target hardware. All works as expected. I loaded MFSystemTest solution and can specify my device as the target and either a single test or a list of tests and it runs as expected. I could not load MFSystemTest and specify the Emulator in the project settings and have that run any test(s) correctly. It would be nice to fix this, but given the automated testing will not require this, and its only useful for debugging MFSystemTest, it is not critical at this time. I have tried opening a couple of solutions and switching to Release build and they deploy and run as expected. Please expand what you mean by the Test SDK? |
The test kit is a separate MSI that adds the tests and test runner to an SDK only installation. Surprised, so was I! (See #451 ) so changes to test projects need to be validated on an installation that only has the SDK + test kit install (e.g. no GIT repo) |
Summary: I have now tested with the Test SDK too. Works as expected within the limitations of the current Test SDK. Details: Clean machine with Windows 10 and VS2015 installed. No GitHUB repository. Test SDK Readme says there should be a cmd prompt installed, but I could not find it in the program list. Opened regular cmd.exe and changed to the folder containing the RunTests.exe. At this point if you use RunTests.exe with the default options to run all the tests on the Emulator, it will fail immediately with a file access violation as it attempts to open/write files in the Tests\Results folder, which is by default installed under Program Files (x86). I then changed file permissions to grant all users Full Control to the Test and lower folders. RunTests.exe will then run as expected and ultimately produce the Results in the default browser. When I ran the tests, most test passed. A few failed. These also fail when run individually on a system with GitHub repository. I did not review why they failed in detail as the failure is not due to the Test SDK setup, but inherent in the actual test. . To load test solutions in VS2015, need to install support for NetMF in VS2015. VS2015 will tell you it has rebuilt a test if you ask it to do a RebuildAll, but it has not. Would need to add BuildEnv.props and the files under Framework\IDE\Targets to what is installed with the Test SDK. Who uses the Test SDK? Does it still need to be maintained? I ask it causes RunTests.exe to have logic in it that uses the Registry to store values that complicate the code's logic. ie. in some places it uses environment variables, in others the system Registry. It may be better to change the approach to use the RunTests.config file.. |
I will update the pull request as I have discovered one of the tests needs to refer to v4.5 as it is a PC application. |
"Who uses the Test SDK?" At this point, basically no-one as we, embarrassingly, didn't know it was a separate MSI. (The reasons we didn't know are long an convoluted and ultimately irrelevant). The test are useful so for the moment we'll maintain it as-is. However, it probably should get merged into the next release of the SDK - or... better yet, rolled into a NUGET package along with VS test explorer extensibility such that you can create unit tests and run them from within VS, fully integrated into the existing test infrastructure of VS. That's a relatively isolated activity from much of the rest of thoughts/plans for future features in NETMF so anyone interested in taking that on, feel free to speak up. |
This PR is waiting on the update mentioned by @doingnz on May 24 |
ping! to @doingnz any plan to update? If no update is received this PR will be closed |
I am finalizing a release of a product using Net MF 4.4. |
This pull request implements the suggestion in #269 to import BuilEnv.props into the Test project files. I great suggestion that makes the Unit Tests far more accessible and user friendly. Hopefully more people will now use them and extend them.
By importing the BuildEnv.props into the C# projects, it is possible to open the solution file directly by double clicking on the .sln file in Windows File Explorer, and then build and Run or Debug the test(s) in the emulator or on native Hardware.
Unit testing by running RunTests.exe with test lists etc remains the same and one can target the emulator or a device as we currently do.
It is possible to open the MFTestSystem.sln file and run it directly too. Modify the command line parameters in the project settings to specify the device to run the tests on. It is possible to single step and debug MFTestSystem.
There is one caveat. Currently I cannot open MFTestSystem.sln and specify the Emulator as the test target and successfully run working unit tests. Specifying a device as a target works as expected with the test deployed to the target device and executed. However, if the Emulator is specified, then the test does not run correctly. It is as is the emulator process terminates early and incomplete XML test results are received, and the test is reported as failed There must be a simple fix for this that I cannot see given RunTests.exe can specify the emulator when run from the command line with the correct behavior against the emulator. .