Testing

Nick Crookston, USFS; Don Robinson, ESSA Technologies, June 2015

Introduction

This Wiki entry provides a brief overview of the testing scheme we have developed. There will be changes in this protocol as the development of open-fvs progresses. Your input is welcome.

Getting the Necessary Tools

You will need Administrator privileges to install the necessary tools, so ensure you have that before proceeding.

The tests make use of standard Unix utilities: make, echo, grep, diff and rm, so carrying out the tests on Unix platforms requires no additional tools. Carrying out tests on Windows systems requires versions of these utilities that have been ported to Windows. There are two packages of free utilities that you may wish to install.

MinGW/MSys option

The first package is recommended if you are building FVS with the 32-bit MinGW (or Visual Studio) toolchain, and is available as an optional component of the suite of MinGW utilities. Follow the steps shown here to download, install and use the MinGW installer (mingw-get-setup.exe). In addition to the compiling tools, select "Basic Setup" and then choose the msys-base components.

This will download and install the additional utilities to the MinGW installation directory.

You will then need to modify the Windows PATH variable to make use of the new utilities, adding this directory:

C:\MinGW\msys\1.0\bin

One way to do this is to right-click the My Computer icon on the Windows desktop and select Properties. Click the Advanced tab, then the Environment Variables button. Locate PATH or Path from either the User variables or System variables, then click Edit. At the end of the current set of PATH directories add "; C:\MinGW\msys\1.0\bin". Then click OK buttons until you get back to the desktop.

Finally, you will need to disable this file

C:\MinGW\msys\1.0\bin\sh.exe

by renaming it to something like sh.disable.exe; otherwise Cmake will not work. The 32-bit Msys utilities can be downloaded and installed (leaving out the 32-bit compiler tools), to provide utilities to enable testing on 64-bit systems. You will need to enable/disable the sh.exe as you switch from a build-focus to a test-focus. Such is life when you use free tools.

Rtools option

The second package is recommended if you are already using rTools to build FVS; in which case the utilities are installed already. See the Build Process Using Rtools page for more information.

Running Tests

There is a tests directory under the trunk. It contains a directory for each FVS program that can be built in the open-fvs system. It also contains a directory called ODBC which contains sub directories for testing two databases, Access and SQLite3.

Inside each sub directory there is a makefile that runs all the tests that are stored in that directory. To run the tests, simply run make:

make all

The each of the tests in the make file is run generally using the following steps:

1. An FVS program is run using an input file. Sometimes the test is run using a response file to answer the traditional prompts and sometimes it is run using command line arguments to specify the keyword file or other arguments.
2. grep may be used to find specific lines in the output that will be compared with saved correct output.
3. diff is run to compare the newly created output with output that is saved and the differences are displayed.

Correct Output

The correct output for each test is stored in a file that has .save as its suffix. Note that this output has been deemed correct by the team member who committed the .save file the repository. It is important to understand that this output was run on one of the supported computer systems and output you get using a different system may be equally correct but different. Review the magnitude of the differences and decide for yourself if the tests pass or that an error has occurred that needs to be addressed.

Creating Tests

A step in the development process is to add to the test library. There are a few steps to follow:

1. Inspect the current tests, pick a pattern to follow (there are several approaches), and construct your tests using what you consider to be the best approach. Be thrifty, these tests will become part of the routine set and will be run thousands of times. But don't be too thrifty, include tests that should be run!
2. Add entries to the makefile as needed; build new makefiles if needed.
3. Run your tests and establish the correct output. Rename the output file by adding .save as the suffix.
4. Commit the new tests and .save file to the repository. Remember to commit new versions of the .save file if you make changes in FVS that result in new correct output.

Operating System-Dependent Tests

• Tests which incorporate input or output using an Access data base (in tests/ODBC and in some variants) rely on having MS Office installed, and on having 32- or 64-bit versions of FVS which correspond with 32- or 64-bit version of MS Office. It should be possible to test 32-bit versions of FVS/Office installed on a 64-bit OS, but not the reverse.
• Tests of the SQLite3 data base (in tests/ODBCand in some variants) rely on having SQLite3 ODBC drivers installed on your system. Installation files for a variety of Operating Systems can be found at SQLite ODBC Driver.