-
Notifications
You must be signed in to change notification settings - Fork 0
TestSuite
_Reliability in the face of complexity_ is the real challenge of software development, as discussed in this paper.
The input files are in the directories below tsuite in the distribution.
The output is in [ftp://gradj.pa.uky.edu/gary/cloudy_gold/tsuite_output/ this] ftp directory.
There are four directories below tsuite in the distribution.
These contain different types of tests.
Those in the auto directory are run every night in Lexington.
These are a large number of simulations that test some aspect of the physics.
The tests in the slow directory take about three days and are continuously running on machines here.
The directories under programs contain programs that call Cloudy as a subroutine.
Finally the crash directory contains a series of tests which should cause the code to
crash if it has been compiled correctly.
Each directory contains several perl scripts and html files.
The readme_*.htm files describe the contents of each directory.
The file doc_tsuite.htm documents the purpose of each of the tests.
It could be studied to see how to set up the code.
The tests should not be placed in a directory which contained an older version until the old version is first deleted. The names of the test cases change from time to time and placing the current version of the suite on top of an old one will not overwrite all the old ones. This will cause problems when the test suite is run. Either start with a new directory or first delete the contents of the directory.
The test suite includes a Perl script to run all the test cases and then check for problems. It is called run_parallel.pl. It will use multiple cores to run several test suite cases in parallel, thus considerably reducing the amount of wallclock time needed to run the test suite. The syntax is:
run_parallel.pl [ <sub-dir> ] [ <#cpu> ] (before C17)
run_parallel.pl [ <sub-dir> ] [ <#cpu> ] [ <list of input scripts> ] (C17 and later)
By default, run_parallel.pl will assume that the Cloudy executable was built in the source directory, use all available cores on your system (provided it can determine that number -- this should work on most systems) and run all the scripts in the test suite. If you built Cloudy in one of the subdirectories, you should add the directory name as a first parameter. If you want to use less cores, or the number was determined incorrectly, simply add the number of cores that should be used as a second parameter. Some typical examples are:
run_parallel.pl (cloudy.exe in source directory, use all available cores)
run_parallel.pl sys_icc 3 (cloudy.exe in sys_icc directory, use 3 cores)
run_parallel.pl . 2 (cloudy.exe in source directory, use 2 cores)
The "." in the last example needs to be used if you built Cloudy in the source directory and you want to override the default number of cores. Providing a higher number of cores than are physically available will not speed up execution (the parallel execution is nearly 100% efficient), but will increase the risk of running out of memory. If you built Cloudy in a non-standard location, you need to provide the full path to the binary, e.g.
run_parallel.pl /path/to/cloudy.exe
Since C17 you can run only a subset of all the available scripts, e.g. like this:
run_parallel.pl . 8 blr.in*
You can use all shell wildcard characters here. This example will only run input scripts starting with blr. Note that in this case it is mandatory to supply the directory and the number of cores, even if you want those at the default values.
_BEWARE: The run_parallel.pl script will clean out the test suite_ (i.e., it calls clean_tsuite.pl) before running the test suite to remove any files from a previous run. Make sure that you saved any relevant output from a previous run of the test suite before executing run_parallel.pl! Don't put any of your own files in the test suite directories, create a separate directory for that. Alternatively, since C17 you can use the script rerun_parallel.pl which works exactly as shown above, except that it does not call clean_tsuite.pl. This is especially useful if you want to rerun a specific subset of the input scripts, but want to keep the output of the other scripts.
Starting with the C08 release there are 3 incarnations of the run_parallel.pl script: tsuite/auto/run_parallel.pl and tsuite/slow/run_parallel.pl run the auto or slow test suite only, while tsuite/run_parallel.pl runs both test suites combined. The same applies to rerun_parallel.pl since C17.
If running the run_parallel.pl fails with a message similar to
Undefined subroutine &RunService::initialize called at ./run_parallel.pl line 16
you may need to install the perl Math::BigInt module. Mitchell Revalski discovered this problem on Cygwin.
If you want to clean up the test suite directory after a run, you can type the command clean_tsuite.pl, which will delete all the test suite output. This is useful if diskspace is at a premium. The test suite output is surprisingly bulky! The command only works in the local directory, so you need to change directory to the test suite directory first before entering the command.
Starting with the C08 release there are 3 incarnations of the clean_tsuite.pl script: tsuite/auto/clean_tsuite.pl and tsuite/slow/clean_tsuite.pl clean the auto or slow test suite only, while tsuite/clean_tsuite.pl cleans both test suites.
A large code must be completely checked every time anything is changed.
The code uses extensive self-checking during a calculation to insure that the results are valid.
The monitor commands in the input files tell the code what answer to expect
based either on analytical or previous results.
This allows the code to confirm that it has found the correct answer.
A distinctive string is printed if the right answer is not obtained.
The Perl script checkall.pl will examine all output files to search for failed monitors,
other problems, or crashes.
These would indicate a problem.
The monitor commands have nothing to do with the simulation or the astrophysics,
but instead provide a way for the code to implement these automatic self-checks.
The Perl script tests_remove_monitors.pl will remove all the monitor commands
if you wish to reuse these scripts for other purposes.
Problems will be announced at the end of the calculation by a line that includes the string PROBLEM, DISASTER or BOTCHED MONITORS. If the code crashes then the normal end-of-calculation string will not be printed.
The test distribution includes the Perl script checkall.pl that will check for problems once the test cases have been computed. Run this script and notice what it says. If you use run_parallel.pl this script will be called automatically after the test suite finished.
The stellar atmosphere test cases stars_*.in require the compiled stellar atmosphere files.
These will not succeed if you did not install the stellar atmospheres. This is OK if you don't need those.
Starting with the C08 release, the checkall.pl script will quietly skip sims that failed because the stellar atmosphere file was not installed and list them in the file skip.txt instead.
The programs directory contains a number of programs that use Cloudy as a subroutine. Each program lives in its own directory, and is accompanied by a readme file, which explains its purpose. Scripts to compile and/or run each of these programs by themselves or in succession are provided.
The directory also contains a template C++ program as an example of using Cloudy as a subroutine. Users are encouraged to modify this template program to meet their needs. A script is provided to guide the compilation of the template and its linking against the Cloudy library. Users are advised to understand the steps that are necessary for linking against Cloudy, and in particular the role the macro SYS_CONFIG plays in this process. Details may be found in readme_template.txt.
Only calls from C++ programs are currently supported.
Next step, apply any HotFixes.
Return to the StepByStep instructions.