From 93ce9f91f49633fada54b1109524b307a856f20d Mon Sep 17 00:00:00 2001 From: Scott Jones Date: Fri, 1 Sep 2023 16:58:36 -0700 Subject: [PATCH 1/5] updated and added detail to test execution --- UITests/README.md | 72 ++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 58 insertions(+), 14 deletions(-) diff --git a/UITests/README.md b/UITests/README.md index 5bf12c88b..f9d1ff848 100644 --- a/UITests/README.md +++ b/UITests/README.md @@ -1,29 +1,73 @@ # UI Tests with WinAppDriver -This UI Test repository of collection of test scenarios that covers basic interactions with [Universal Windows Platform controls (UI elements)](https://docs.microsoft.com/en-us/windows/uwp/controls-and-patterns/). Use this as a reference on how to interact with certain UWP controls you have in your application. +This UI Test repository of collection of WinAppDriver-based test scenarios that cover basic interactions with WinUI 3 controls. Note: the [Universal Windows Platform controls](https://docs.microsoft.com/en-us/windows/uwp/controls-and-patterns/) reference can be used, as the WinUI 3 controls are very similar to the UWP controls. -The test scenarios are written to test the controls in the Xaml Controls Gallery app, which contains all UWP Xaml controls. The Xaml Controls Gallery app needs to be installed on the machine where you are running your tests. Follow the instructions in the root README to install the app. +The test scenarios are written to test the controls in the Xaml Controls Gallery app. The procedure below outlines the steps needed to build, deploy, and stage the Xaml Controls Gallery and WinAppDriver to run the UI Tests. These manual steps simulate what azure-pipelines.yml does in pipeline test runs. +## Build and install Xaml Controls Gallery -## Requirements +1. Generate the test signing certificate: -- Windows 10 PC with the latest Windows 10 version (Version 1809 or later) -- Microsoft Visual Studio 2017 or later -- [Xaml Controls Gallery](../../XamlControlsGallery/XamlControlsGallery.sln) built and deployed +```shell + PS> .\build\GenerateTestPfx.ps1 +``` +1. Build and publish the Xaml Controls Gallery from the command line, e.g.: -## Getting Started +```shell + dotnet.exe publish WinUIGallery\WinUIGallery.sln /p:AppxPackageDir=AppxPackages\ /p:platform=x64 /p:PublishProfile=./WinUIGallery/Properties/PublishProfiles/win10-x64.pubxml +``` -1. Open `UITests.sln` in Visual Studio -2. Select **Test** > **Windows** > **Test Explorer** -3. Select **Run All** on the test pane or through menu **Test** > **Run** > **All Tests** +1. Locate the Xaml Controls Gallery package output folder from above and deploy for testing: -> Once the project is successfully built, you can use the **TestExplorer** to pick and choose the test scenario(s) to run +```shell + PS> .\WinUIGallery\AppxPackages\WinUIGallery.Desktop_Test\Install.ps1 +``` +## Install and execute WinAppDriver -## Adding/Updating Test Scenario +1. Download and install the latest version of WinAppDriver from [here](https://github.com/microsoft/WinAppDriver/releases) + +1. Run WinAppDriver (required for running tests on cmdline or in VS): + +```shell + C:\Program Files (x86)\Windows Application Driver>WinAppDriver.exe +``` + +## Build and run/debug Tests + +1. Build UITests: + +```shell + >msbuild UITests\UITests.sln +``` + +1. Run test cases built above on command line: + +```shell + vstest.console.exe D:\git\WinUI-Gallery\UITests\bin\x64\Debug\net7.0\UITests.dll +``` + +1. Debug test cases in Visual Studio + * Open `UITests.sln` in Visual Studio + * Select **Test** > **Windows** > **Test Explorer** + * Select **Run All** on the test pane or through menu **Test** > **Run** > **All Tests** + + Once the project is successfully built, you can use the **TestExplorer** to pick and choose the test scenario(s) to run/debug + +## Test output + +In either scenario above (cmdline or VS), if the tests run successfully, you should see: + +* WinAppDriver console spew indicating successful launch of the test app +* The Xaml Controls Gallery launch and run with automated UI interactions +* vstest.console.exe spew (if cmdline) showing test case status + + +## Adding/Updating Test Scenarios Please follow the guidelines below to maintain test reliability and reduce test execution time: + 1. Provide a complete set of interactions (if applicable) for each new control -2. Aim for simple and reliable scenario using the least amount of test steps -3. Reuse existing application session when possible to reduce unnecessary application re-launching +1. Aim for simple and reliable scenario using the least amount of test steps +1. Reuse existing application session when possible to reduce unnecessary application re-launching From 1e1efb8087a4efd4ff9b3b396547e5ad746af620 Mon Sep 17 00:00:00 2001 From: Scott Jones Date: Wed, 6 Sep 2023 09:10:52 -0700 Subject: [PATCH 2/5] Update UITests/README.md Co-authored-by: Marcel Wagner --- UITests/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/UITests/README.md b/UITests/README.md index f9d1ff848..e26f7fcb6 100644 --- a/UITests/README.md +++ b/UITests/README.md @@ -18,7 +18,7 @@ The test scenarios are written to test the controls in the Xaml Controls Gallery dotnet.exe publish WinUIGallery\WinUIGallery.sln /p:AppxPackageDir=AppxPackages\ /p:platform=x64 /p:PublishProfile=./WinUIGallery/Properties/PublishProfiles/win10-x64.pubxml ``` -1. Locate the Xaml Controls Gallery package output folder from above and deploy for testing: +1. Locate the WinUI 3's Gallery package output folder from above and deploy for testing: ```shell PS> .\WinUIGallery\AppxPackages\WinUIGallery.Desktop_Test\Install.ps1 From 3e797098766d56bfdbabc53a65c41e8e12090c6a Mon Sep 17 00:00:00 2001 From: Scott Jones Date: Wed, 6 Sep 2023 11:37:00 -0700 Subject: [PATCH 3/5] PR feedback --- UITests/README.md | 42 ++++++++++++++++++++++++------------------ 1 file changed, 24 insertions(+), 18 deletions(-) diff --git a/UITests/README.md b/UITests/README.md index e26f7fcb6..a5f270797 100644 --- a/UITests/README.md +++ b/UITests/README.md @@ -2,9 +2,11 @@ This UI Test repository of collection of WinAppDriver-based test scenarios that cover basic interactions with WinUI 3 controls. Note: the [Universal Windows Platform controls](https://docs.microsoft.com/en-us/windows/uwp/controls-and-patterns/) reference can be used, as the WinUI 3 controls are very similar to the UWP controls. -The test scenarios are written to test the controls in the Xaml Controls Gallery app. The procedure below outlines the steps needed to build, deploy, and stage the Xaml Controls Gallery and WinAppDriver to run the UI Tests. These manual steps simulate what azure-pipelines.yml does in pipeline test runs. +The test scenarios are written to test the controls in the WinUI 3 Gallery app. The procedure below outlines the steps needed to build and deploy WinUI 3 Gallery and WinAppDriver to run the UI Tests. These steps mirror what azure-pipelines.yml does in pipeline runs. -## Build and install Xaml Controls Gallery +## Deploy WinUI 3 Gallery + +The easiest way to deploy WinUI 3 Gallery for unit test execution is to simply build WinUIGallery\WinUIGallery.sln and F5 deploy it from within Visual Studio. Alternatively, the following commands can be used for automation. 1. Generate the test signing certificate: @@ -12,23 +14,25 @@ The test scenarios are written to test the controls in the Xaml Controls Gallery PS> .\build\GenerateTestPfx.ps1 ``` -1. Build and publish the Xaml Controls Gallery from the command line, e.g.: +1. Build and publish the WinUI 3 Gallery from the command line, e.g.: ```shell dotnet.exe publish WinUIGallery\WinUIGallery.sln /p:AppxPackageDir=AppxPackages\ /p:platform=x64 /p:PublishProfile=./WinUIGallery/Properties/PublishProfiles/win10-x64.pubxml ``` -1. Locate the WinUI 3's Gallery package output folder from above and deploy for testing: +1. Locate the WinUI 3 Gallery package output folder from above and deploy for testing: ```shell PS> .\WinUIGallery\AppxPackages\WinUIGallery.Desktop_Test\Install.ps1 ``` -## Install and execute WinAppDriver +## Run WinAppDriver + +The test runner (vstest.console.exe and VS Test Explorer) should automatically launch WinAppDriver. Alternatively, it can be launched manually to observe diagnostic output as follows. 1. Download and install the latest version of WinAppDriver from [here](https://github.com/microsoft/WinAppDriver/releases) -1. Run WinAppDriver (required for running tests on cmdline or in VS): +1. Run WinAppDriver: ```shell C:\Program Files (x86)\Windows Application Driver>WinAppDriver.exe @@ -36,6 +40,16 @@ The test scenarios are written to test the controls in the Xaml Controls Gallery ## Build and run/debug Tests +The easiest way to run/debug the UI tests is with Visual Studio, as follows: + + * Open `UITests\UITests.sln` in Visual Studio + * Select **Test** > **Windows** > **Test Explorer** + * Select **Run All** on the test pane or through menu **Test** > **Run** > **All Tests** + + Once the project is successfully built, you can use the **Test Explorer** to choose test scenarios to run/debug + +Alternatively, the following commands can be used for automation: + 1. Build UITests: ```shell @@ -48,21 +62,13 @@ The test scenarios are written to test the controls in the Xaml Controls Gallery vstest.console.exe D:\git\WinUI-Gallery\UITests\bin\x64\Debug\net7.0\UITests.dll ``` -1. Debug test cases in Visual Studio - * Open `UITests.sln` in Visual Studio - * Select **Test** > **Windows** > **Test Explorer** - * Select **Run All** on the test pane or through menu **Test** > **Run** > **All Tests** - - Once the project is successfully built, you can use the **TestExplorer** to pick and choose the test scenario(s) to run/debug - ## Test output In either scenario above (cmdline or VS), if the tests run successfully, you should see: -* WinAppDriver console spew indicating successful launch of the test app -* The Xaml Controls Gallery launch and run with automated UI interactions -* vstest.console.exe spew (if cmdline) showing test case status - +* The WinUI 3 Gallery launch and run with automated UI interactions +* vstest.console.exe spew or Test Explorer indications of test case status +* Optionally, WinAppDriver console spew indicating successful launch of the test app ## Adding/Updating Test Scenarios @@ -70,4 +76,4 @@ Please follow the guidelines below to maintain test reliability and reduce test 1. Provide a complete set of interactions (if applicable) for each new control 1. Aim for simple and reliable scenario using the least amount of test steps -1. Reuse existing application session when possible to reduce unnecessary application re-launching + From 4756f829cf35b645cd01a6b61bd904f5e35e21fc Mon Sep 17 00:00:00 2001 From: Scott Jones Date: Thu, 7 Sep 2023 12:15:10 -0700 Subject: [PATCH 4/5] PR feedback --- UITests/README.md | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/UITests/README.md b/UITests/README.md index a5f270797..164927cba 100644 --- a/UITests/README.md +++ b/UITests/README.md @@ -1,12 +1,12 @@ # UI Tests with WinAppDriver -This UI Test repository of collection of WinAppDriver-based test scenarios that cover basic interactions with WinUI 3 controls. Note: the [Universal Windows Platform controls](https://docs.microsoft.com/en-us/windows/uwp/controls-and-patterns/) reference can be used, as the WinUI 3 controls are very similar to the UWP controls. +The UI Test repository is a collection of WinAppDriver-based test scenarios that cover basic interactions with WinUI 3 controls. Note: the [Universal Windows Platform controls](https://docs.microsoft.com/en-us/windows/uwp/controls-and-patterns/) reference can be used, as the WinUI 3 controls are very similar to the UWP controls. The test scenarios are written to test the controls in the WinUI 3 Gallery app. The procedure below outlines the steps needed to build and deploy WinUI 3 Gallery and WinAppDriver to run the UI Tests. These steps mirror what azure-pipelines.yml does in pipeline runs. ## Deploy WinUI 3 Gallery -The easiest way to deploy WinUI 3 Gallery for unit test execution is to simply build WinUIGallery\WinUIGallery.sln and F5 deploy it from within Visual Studio. Alternatively, the following commands can be used for automation. +The easiest way to deploy the WinUI 3 Gallery for unit test execution is to simply build WinUIGallery\WinUIGallery.sln and F5 deploy it from within Visual Studio. Alternatively, the following commands can be used for automation. 1. Generate the test signing certificate: @@ -28,14 +28,14 @@ The easiest way to deploy WinUI 3 Gallery for unit test execution is to simply b ## Run WinAppDriver -The test runner (vstest.console.exe and VS Test Explorer) should automatically launch WinAppDriver. Alternatively, it can be launched manually to observe diagnostic output as follows. +The test runner (vstest.console.exe and VS Test Explorer) should automatically launch WinAppDriver. Alternatively, it can be launched manually to observe diagnostic output as follows: 1. Download and install the latest version of WinAppDriver from [here](https://github.com/microsoft/WinAppDriver/releases) 1. Run WinAppDriver: ```shell - C:\Program Files (x86)\Windows Application Driver>WinAppDriver.exe + C:\Program Files (x86)\Windows Application Driver\WinAppDriver.exe ``` ## Build and run/debug Tests @@ -46,20 +46,24 @@ The easiest way to run/debug the UI tests is with Visual Studio, as follows: * Select **Test** > **Windows** > **Test Explorer** * Select **Run All** on the test pane or through menu **Test** > **Run** > **All Tests** - Once the project is successfully built, you can use the **Test Explorer** to choose test scenarios to run/debug +Once the project is successfully built, you can use the **Test Explorer** to choose test scenarios to run/debug Alternatively, the following commands can be used for automation: 1. Build UITests: ```shell + >dotnet build UITests\UITests.sln + --or-- >msbuild UITests\UITests.sln ``` 1. Run test cases built above on command line: ```shell - vstest.console.exe D:\git\WinUI-Gallery\UITests\bin\x64\Debug\net7.0\UITests.dll + >dotnet test .\UITests\UITests.csproj + --or-- + >vstest.console.exe .\UITests\bin\x64\Debug\net7.0\UITests.dll ``` ## Test output From c23330ff51c723f83bbc34c5baa81e345fec39ce Mon Sep 17 00:00:00 2001 From: Scott Jones Date: Thu, 7 Sep 2023 12:52:41 -0700 Subject: [PATCH 5/5] PR feedback --- UITests/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/UITests/README.md b/UITests/README.md index 164927cba..075757db3 100644 --- a/UITests/README.md +++ b/UITests/README.md @@ -17,7 +17,7 @@ The easiest way to deploy the WinUI 3 Gallery for unit test execution is to simp 1. Build and publish the WinUI 3 Gallery from the command line, e.g.: ```shell - dotnet.exe publish WinUIGallery\WinUIGallery.sln /p:AppxPackageDir=AppxPackages\ /p:platform=x64 /p:PublishProfile=./WinUIGallery/Properties/PublishProfiles/win10-x64.pubxml + >dotnet.exe publish WinUIGallery\WinUIGallery.sln /p:AppxPackageDir=AppxPackages\ /p:platform=x64 /p:PublishProfile=./WinUIGallery/Properties/PublishProfiles/win10-x64.pubxml ``` 1. Locate the WinUI 3 Gallery package output folder from above and deploy for testing: @@ -35,7 +35,7 @@ The test runner (vstest.console.exe and VS Test Explorer) should automatically l 1. Run WinAppDriver: ```shell - C:\Program Files (x86)\Windows Application Driver\WinAppDriver.exe + >C:\Program Files (x86)\Windows Application Driver\WinAppDriver.exe ``` ## Build and run/debug Tests