diff --git a/README.md b/README.md index 5c1489975..4a04e8885 100644 --- a/README.md +++ b/README.md @@ -18,17 +18,27 @@ Testrun creates an isolated and controlled network environment on a linux machin Two modes are supported by Testrun: -1) Automated testing +
+ + Automated testing + Once the device has become operational (steady state), automated testing of the DUT (device under test) will begin. Containerized test modules will then execute against the device, one module at a time. Once all test modules have been executed, a report will be produced - presenting the results. +
-2) Lab network +
+ + + Lab network + When manual testing or configuration changes are required, Testrun will provide the network and some tools to assist an engineer performing the additional testing. This reduces the need to maintain a separate but identical lab network. Testrun will take care of packet captures and logs for each network service for further debugging. +
+ ## Minimum requirements :computer: ### Hardware - - PC running Ubuntu LTS 20.04 or 22.04 (laptop or desktop) + - PC running Ubuntu LTS 20.04, 22.04 or 24.04 (laptop or desktop) - 2x USB ethernet adapter (One may be built in ethernet) - Internet connection ### Software @@ -43,10 +53,10 @@ Once you have met the hardware and software requirements, you can get started wi Testrun will constantly evolve to further support end-users by automating device network behaviour against industry standards. For further information on upcoming features, check out the [Roadmap](docs/roadmap.pdf). ## Accessibility :busts_in_silhouette: -We are proud of our tool and strive to provide an enjoyable experience for all of our users. Testrun goes through rigorous accessibility testing at each release. You can read more about [Google and Accessibility here](https://www.google.co.uk/accessibility). You are welcome to submit a new issue and provide feedback on our implementations. +We are proud of our tool and strive to provide an enjoyable experience for all of our users. Testrun goes through rigorous accessibility testing at each release. You can read more about [Google and Accessibility here](https://www.google.co.uk/accessibility). You are welcome to submit a new issue and provide feedback on our implementations. To find out how Testrun implements accessibility features, you can view a [short video here](docs/ui/accessibility.mp4). ## Issue reporting :triangular_flag_on_post: -If the application has come across a problem at any point during setup or use, please raise an issue under the [issues tab](https://github.com/auto-iot/test-run/issues). Issue templates exist for both bug reports and feature requests. If neither of these are appropriate for your issue, raise a blank issue instead. +If the application has come across a problem at any point during setup or use, please raise an issue under the [issues tab](https://github.com/google/testrun/issues). Issue templates exist for both bug reports and feature requests. If neither of these are appropriate for your issue, raise a blank issue instead. ## Contributing :keyboard: The contributing requirements can be found in [CONTRIBUTING.md](CONTRIBUTING.md). In short, checkout the [Google CLA](https://cla.developers.google.com/) site to get started. @@ -74,8 +84,4 @@ The contributing requirements can be found in [CONTRIBUTING.md](CONTRIBUTING.md) 4) Can I run Testrun on a virtual machine? - Testrun can be virtualized if the 2x ethernet adapters are passed through to a VirtualBox VM as a USB device rather than managed network adapters. A full guide will be provided once virtualization of Testrun has been fully tested. - - 5) Can I connect multiple devices to Testrun? - - In short, Yes you can. The way in which multiple devices could be tested simultaneously is yet to be decided. However, if you simply want to add field/peer devices during runtime (even another laptop performing manual testing) then you may connect the USB ethernet adapter to an unmanaged switch. + Testrun can be virtualized if the 2x ethernet adapters are passed through to a VirtualBox VM as a USB device rather than managed network adapters. You can view the guide to working on a [virtual machine here](docs/virtual_machine.md). diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 000000000..96eb32223 --- /dev/null +++ b/docs/README.md @@ -0,0 +1,18 @@ +Testrun logo + + +## Contents + + - [Get Started](get_started.md) + - [Network](network/README.md) + - [Network Overview](network/README.md) + - [How to identify network interfaces](network/identify_interfaces.md) + - [Addresses](network/addresses.md) + - [Add a new network service](network/add_new_service.md) + - [Testing](test/README.md) + - [Test modules](test/modules.md) + - [Test statuses](test/statuses.md) + - [Development](dev/README.md) + - [Running on a virtual machine](virtual_machine.md) + - [Accessibility](ui/accessibility.mp4) + - [Roadmap](roadmap.pdf) diff --git a/docs/configure_device.md b/docs/configure_device.md index 9eefcd866..1db7155be 100644 --- a/docs/configure_device.md +++ b/docs/configure_device.md @@ -1,4 +1,6 @@ -# Device Configuration +Testrun logo + +## Device Configuration (Deprecated) The device configuration file allows you to customize the testing behavior for a specific device. This file is located at `local/devices/{Device Name}/device_config.json`. Below is an overview of how to configure the device tests. diff --git a/docs/get_started.md b/docs/get_started.md index 07a7d7853..f9a2aa2f8 100644 --- a/docs/get_started.md +++ b/docs/get_started.md @@ -1,6 +1,9 @@ -# Getting Started +Testrun logo -It is recommended that you run Testrun on a standalone machine running a fresh-install of Ubuntu 22.04.3 LTS. + +## Getting Started + +It is recommended that you run Testrun on a standalone machine running a fresh install of Ubuntu 20.04, 22.04 or 24.04 LTS (laptop or desktop). ## Prerequisites @@ -12,6 +15,8 @@ Before starting with Testrun, ensure you have the following hardware: - 2x USB Ethernet adapter (one may be a built-in Ethernet port) - Internet connection +![Visual representation of setup](setup/visual.png) + **NOTE: Running in a virtual machine? Checkout the virtual machine documentation [here](/docs/virtual_machine.md).** ### Software @@ -34,17 +39,18 @@ However, to achieve a compliant test outcome, your device must be configured cor ## Installation -1. Download the latest version of the Testrun installer from the [releases page](https://github.com/google/test-run/releases) +1. Download the latest version of the Testrun installer from the [releases page](https://github.com/google/testrun/releases) 2. Open a terminal and navigate to location of the Testrun installer (most likely your downloads folder) 3. Install the package using ``sudo apt install ./testrun*.deb`` - Testrun will be installed under the /usr/local/testrun directory. - - Testing data will be available in the ``local/devices/{device}/reports`` folders - - Additional configuration options are available in the ``local/system.json`` file + - Testing data will be available in the ``local/devices/{device}/reports`` folders + + **NOTE: Local CA certificates should be uploaded within Testrun to run TLS server testing** - **NOTE: Place your local CA certificate in local/root_certs (any name with a .crt extension) to perform TLS server tests** + ![Installing Testrun in terminal window](setup/install.gif) ## Start Testrun @@ -53,8 +59,8 @@ However, to achieve a compliant test outcome, your device must be configured cor - Connect the other USB Ethernet adapter directly to the IoT device you want to test using an ethernet cable. Some things to remember: + - Both adapters should be disabled in the host system (IPv4, IPv6 and general). You can do this by going to Settings > Network - The device under test should be powered off until prompted - - Both adapters should be disabled in the host system (IPv4, IPv6 and general). You can do this by going to Settings > Network - Struggling to identify the correct interfaces? See [this guide](network/identify_interfaces.md). 2. Start Testrun. @@ -111,7 +117,7 @@ If you encounter any issues or need assistance, consider the following: - Refer to the Testrun documentation or ask for assistance in the issues page: https://github.com/google/testrun/issues # Reviewing -Once you have completed a test attempt, you may want to review the test report provided by Testrun. For more information about what Testrun looks for when testing, and what the output means, take a look at the testing documentation: [Testing](/docs/test/index.md). +Once you have completed a test attempt, you may want to review the test report provided by Testrun. For more information about what Testrun looks for when testing, and what the output means, take a look at the testing documentation: [Testing](/docs/test/README.md). # Uninstall To uninstall Testrun, use the built-in dpkg uninstall command to remove Testrun correctly. For Testrun, this would be: ```sudo apt-get remove testrun```. diff --git a/docs/network/README.md b/docs/network/README.md index 601ed1349..b5536c30c 100644 --- a/docs/network/README.md +++ b/docs/network/README.md @@ -1,4 +1,7 @@ -# Network Overview +Testrun logo + + +## Network Overview ## Table of Contents 1) Network Overview (this page) @@ -6,7 +9,7 @@ 3) [Addresses](addresses.md) 4) [Add a new network service](add_new_service.md) -Test Run provides several built-in network services that can be utilized for testing purposes. These services are already available and can be used without any additional configuration. +Testrun provides several built-in network services that can be utilized for testing purposes. These services are already available and can be used without any additional configuration. The following network services are provided: diff --git a/docs/network/add_new_service.md b/docs/network/add_new_service.md index 5f7b470cd..7a07e43be 100644 --- a/docs/network/add_new_service.md +++ b/docs/network/add_new_service.md @@ -1,4 +1,6 @@ -# Adding a New Network Service +Testrun logo + +## Adding a New Network Service The Testrun framework allows users to add their own network services with ease. A template network service can be used to get started quickly, this can be found at [modules/network/template](../../modules/network/template). Otherwise, see below for details of the requirements for new network services. diff --git a/docs/network/addresses.md b/docs/network/addresses.md index cbefc84a0..7fa71d716 100644 --- a/docs/network/addresses.md +++ b/docs/network/addresses.md @@ -1,4 +1,6 @@ -# Network Addresses +Testrun logo + +## Network Addresses Each network service is configured with an IPv4 and IPv6 address. For IPv4 addressing, the last number in the IPv4 address is fixed (ensuring the IP is unique). See below for a table of network addresses: diff --git a/docs/network/identify_interfaces.md b/docs/network/identify_interfaces.md index 6baddb6ca..50e62acd3 100644 --- a/docs/network/identify_interfaces.md +++ b/docs/network/identify_interfaces.md @@ -1,4 +1,6 @@ -# Identifying network interfaces +Testrun logo + +## Identifying network interfaces For Testrun to operate correctly, you must select the correct network interfaces within the settings panel of the user interface. There are 2 methods to identify the correct network interfaces: diff --git a/docs/roadmap.pdf b/docs/roadmap.pdf index a3e4dfa0a..9f4599ee1 100644 Binary files a/docs/roadmap.pdf and b/docs/roadmap.pdf differ diff --git a/docs/setup/install.gif b/docs/setup/install.gif new file mode 100644 index 000000000..7d5ef1b2c Binary files /dev/null and b/docs/setup/install.gif differ diff --git a/docs/setup/visual.png b/docs/setup/visual.png new file mode 100644 index 000000000..57f76bffc Binary files /dev/null and b/docs/setup/visual.png differ diff --git a/docs/test/index.md b/docs/test/README.md similarity index 62% rename from docs/test/index.md rename to docs/test/README.md index b440a86f1..19aa691d8 100644 --- a/docs/test/index.md +++ b/docs/test/README.md @@ -1,4 +1,7 @@ -# Testing +Testrun logo + + +## Testing The test requirements that are investigated by Testrun can be found in the [test modules documentation](/docs/test/modules.md). diff --git a/docs/test/modules.md b/docs/test/modules.md index 40c66f29e..7c5851ba4 100644 --- a/docs/test/modules.md +++ b/docs/test/modules.md @@ -1,4 +1,6 @@ -# Test Modules +Testrun logo + +## Test Modules Testrun provides some pre-built test modules for you to use when testing your own device. These test modules are listed below: diff --git a/docs/test/statuses.md b/docs/test/statuses.md index c83259fb3..d196fa4af 100644 --- a/docs/test/statuses.md +++ b/docs/test/statuses.md @@ -1,13 +1,14 @@ -# Test Statuses +Testrun logo + +## Test Statuses Testrun will output the result and description of each automated test. The test results will be one of the following: | Name | Description | What next? | |---|---|---| | Compliant | The device implements the required feature correctly | Nothing | | Non-Compliant | The device does not support the specified requirements for the test | Modify or implement the required functionality on the device | -| Feature Not Present | The device does not implement a feature covered by the test | You may implement the functionality (not required) | +| Feature Not Detected | The device does not implement a feature covered by the test | You may implement the functionality (not required) | | Error | An error occured whilst running the test | Create a bug report requesting additional support to diagnose the issue | -| Skipped | The test has not been executed because a linked test did not produce a compliant result | You may implement the functionality (not required) | ## Test Requirement Testrun also determines whether each test is required for the device to receive an overall compliant result. These rules are: @@ -29,4 +30,4 @@ All required tests are implemented correctly, and all required if applicable tes One or more of the required tests (or required if applicable tests) have produced a non-compliant result. ### Error -One of more of the required tests (or required if applicable tests) have not executed correctly. This does not necessarily indicate that the device is compliant or non-compliant. \ No newline at end of file +One of more of the required tests (or required if applicable tests) have not executed correctly. This does not necessarily indicate that the device is compliant or non-compliant. diff --git a/docs/ui/accessibility.mp4 b/docs/ui/accessibility.mp4 new file mode 100644 index 000000000..1c57ff252 Binary files /dev/null and b/docs/ui/accessibility.mp4 differ diff --git a/docs/virtual_machine.md b/docs/virtual_machine.md index 567d14379..2f029b296 100644 --- a/docs/virtual_machine.md +++ b/docs/virtual_machine.md @@ -1,4 +1,6 @@ -# Virtual Machine +Testrun logo + +## Virtual Machine This guide will provide steps to use Testrun within a virtual machine in virtual Box (VMWare and other providers have not yet been tested). You should use this guide alongside the [Get Started guide](/docs/get_started.md) - only differences will be outlined in this guide.