PySys can be used with Python 3.5/3.6/3.7+ or Python 2.7.
See installation notes in README.rst for more details.
- Removed DEFAULT_STYLESHEET pysys-log.xsl as referenced in XMLResultsWriter, as it cannot be made to work in most modern browsers (e.g. Chrome, Firefox) for security reasons and is not widely used. If you need this functionality, the ability to specify a custom .xsl stylesheet for the XMLResultsWriter is still available as a configuration option in pysysproject.xml.
- The available options for installing PySys have been reworked and modernised. The recommended way to install PySys is by running pip install PySys. A binary .whl wheel is now available for the first time, which is more efficient, reliable and lightweight than other installation methods, and is used by the pip installer. The tar.gz source distribution is still available but is no longer a recommended installation mechanism. The Windows GUI installer is no longer published as this is superseded by the simpler installation experience provided by pip.
- HTML documentation of the PySys API is no longer installed locally by default, but is available on https://pysys-test.github.io/pysys-test website or as a separate zip file available from https://github.com/pysys-test/pysys-test/releases.
- Any custom performance reporter classes created using PySys 1.3.0 and which provided a custom constructor should be updated to include the **kwargs parameter added in this version of PySys, as the old constructor signature is now deprecated. As this API was added in 1.3.0 no other versions are affected.
PySys can now be installed simply by running pip install PySys.
Added optional errors= parameter to pycompat.openfile() to allow easily enabling non-strict handling of unsupported characters for cases where it does not matter (e.g. logging file contents), without needing caller to special-case for Python 2 where no encoding is performed and the parameter is not available. ProcessUser.logFileContents() now uses errors='replace' since for diagnostic logging a best-effort approach to non-ASCII characters is most helpful.
Added __str__ implementations for BaseTest and BaseRunner, which uniquely identify the test (and cycle, in multi-cycle runs), for diagnostic and logging purposes.
Added PerformanceUnit.NANO_SECONDS (with alias ns) for use when measuring the peformance of operations that take less than a second.
Performance reporter classes can now make use of self.runner to access information such as the mode in which the test is running for reporting purposes.
Added boolean IS_WINDOWS constant, since conditionalizing logic for Windows versus all other Operating Systems is very common; this avoids the need for error-prone matching against string literals.
Added skipTest() method, which can be used to avoid running the rest of the execute() or validate() method, if it is not appropriate for the test to execute on this platform/mode.
Fixed bug in which symbols (classes, constants, imports) defined in one run.py could be seen by other run.py files, potentially causing test behaviour to vary based on what other tests had previously run, and/or race conditions seen only during parallel execution. Now every run.py file has its own independent namespace. It is possible some previously passing tests might fail as a result of this change, if they were relying on the buggy behaviour to implicitly import symbols.
Outcome reason now has a suffix specifying how many additional failure outcomes were logged (so if you have a complex test you can see at a glance if there's just one problem to resolve, or 5, or 20!).
- Extended the writers API:
- Added runLogOutput= parameter to the processResult() method of the BaseResultsWriter class so that writers such as the JUnitXMLResultsWriter can include the test output with no loss of unicode character information.
- Added testoutdir= parameter to the setup() method so writers have a way to identify different test runs on the same machine.
- Added runner= parameter to the setup() method so writers have access to the runner instance for reading/modifying configuration settings.
- Added isEnabled() method that can optionally be used by a writer to disable itself based on the environment in which it is running, or to enable itself even when --record isn't specified, which is useful for writers that produce output for a CI system.
Added support for running PySys tests under Travis CI to the sample pysysproject.xml file, including by default only printing run.log output for failed tests, and containing that detailed output within a folded section that can be expanded if needed. To enable this just ensure that the Travis CI writer is enabled in your project configuration file.
Added support for configuring the default encodings to use for common file patterns in the pysysproject.xml configuration, e.g.
<default-file-encoding pattern="*.yaml" encoding="utf-8"/>.
The pysys-examples/pysysproject.xml sample project configuration file now sets utf-8 as the default encoding for XML, json and yaml files, and also for testcase run.log files (though run.log continues to be written in local encoding unless the project file is updated). For more information on this feature, see comments in pysysproject.xml and in ProcessUser.getDefaultFileEncoding().
The default implementation of BaseTest.getDefaultFileEncoding() now delegates to the runner's implementation, allowing customizations to be performed in just one place if neede for both BaseTest and runner class.
Use of print() rather than self.log is a common mistake that results in essential diagnostic information showing up on the console but not stored in run.log. A new project option redirectPrintToLogger can optionally be enabled to instruct PySys to catch output written using print() statements or to sys.stdout and redirect it to the logging framework, so it will show up in run.log. Writers that genuinely need the ability to write directly to stdout should be changed to use pysys.utils.logutils.stdoutPrint.
Added new command line option --printLogs all|failures|none (default value is all) which allows user to avoid the printing of run.log to the stdout console either for all tests, or for tests that pass. This is useful to avoid generating huge amounts of output during large test runs (which can be problematic when stdout is captured by a Continuous Integration system), or to show detailed information only for failing tests which makes it easier for a user to locate the diagnostic information they care about more quickly. The printLogs option is stored in runner.printLogs` and can be changed by custom writer implementations if desired, for example to avoid duplicating information already being printed to stdout by the writer in a different format.
PySys will now automatically enable colored output if there is no color setting in the pysysproject.xml or PYSYS_COLOR environment - provided PySys is running in an interative terminal, and that if running on Windows the colorama library is installed.
Added --threads auto which is equivalent to --threads 0 and provides a clearer way to indicate that PySys will automatically determine how many threads to run tests with based on the number of available CPUs.
As an alternative to the usual pysys.py executable script, it is now also possible to launch PySys using:
python -m pysys
- Fixed startProcess() to add a BLOCKED test outcome when a process fails to start due to a ProcessError, unless ignoreExitStatus=True. Previously this flag only affected non-zero exit codes, resulting in ProcessError failures getting ignored.
- JUnitXMLResultsWriter and XMLResultsWriter now write using UTF-8 encoding rather than local/default encoding, and also include the encoding="utf-8" header in the XML header. Since previously there was no encoding header many tools would have interpreted them as UTF-8 already, and now the behaviour is consistent with that expectation.
- Fix bug in which non-ASCII characters in test outcome reasons could prevent the test log being written to disk if executed in multi-threaded mode. Only affects Python 2.
- Fixed startProcess() to correctly handle passing empty arguments, and arguments containing spaces, quotes and glob characters on Windows. Previously, empty arguments were skipped, and arguments containing spaces were only handled correctly if first character was not a space.
- Fixed rare condition in which performance result reporting would be prevented due to spurious error about resultKey already being used.
- Fixed startProcessMonitor() on Windows to return correct values instead of negative values for large numbers (such as the virtual memory usage).
- Fixed startProcessMonitor() on Windows to take a few seconds instead of a few minutes, by using the Process performance counter to get % Processor Time instead of the Thread counter (the values reported by previous versions were probably not correct, as they only measured CPU for the threads that existed when the process monitor was started).
- Significant improvements to robustness when testing support for international (I18N) characters. This includes implementing fully safe logging of unicode strings (with ? replacements for any unsupported characters) that works regardless of what encoding is in use for stdout and run.log. Also fixed exception when logging unicode characters in Python 2 if a formatter was not configured in pysysproject.xml, by ensuring it is always stored as a unicode character string not a byte string (which used to happen in Python 2 if it was not mentioned in the project config). Fixed logFileContents to more robustly handle files containing I18N/non-ASCII characters.
- Added pysys.writers.replaceIllegalXMLCharacters() utility function, and use it to avoid XMLResultsWriter and JUnitXMLResultsWriter from generating invalid XML if run.log or outcome reason contain characters not permitted by XML. Also ASCII control characters (e.g. coloring instructions from other tools) are now stripped out of all outcome reason strings (including in run.log and non-XML based writers) since such characters are not useful and make summary test results harder to read.
- Fixed bug in which random log lines might not be written to run.log and/or stdout when running tests multi-threaded (as a result of an underlying python bug https://bugs.python.org/issue35185).
Changes affecting compatibility:
- Fixed BaseTest.assertDiff (and filediff) handling of "include" expressions list to filter out lines if no include expressions match (as documented) rather than if any include expressions match. This fix may cause tests to fail that had previously - and incorrectly - passed as a result of all lines being filtered out before the comparison. There is also now a message logged at warn level when every line in a file comparison is filtered out, since in most cases this is not desirable behaviour.
- Changed pysys.py run to return a non-zero exit code if any tests failed, whereas previously it would return 0.
Other fixes and new features:
PySys now provides 'single-source' support for both Python 2.7 and Python 3.x, without the need for the 2to3.py script to be run at installation time for use with Python 3.
Added support for specifying what file encoding is to be used for reading and writing text files (for example in waitForSignal and various assertions). This is especially important for Python 3 where text files are processed using unicode character strings rather than Python 2 byte "str" objects. The encoding can be specified explicitly on individual methods the open files, or globally based on file names or extensions by overriding the new ProcessUser.getDefaultFileEncoding() method. For example, getDefaultFileEncoding could be overridden to specify that .xml files should be treated as UTF-8 by default. If the encoding is not specified explicitly or through getDefaultFileEncoding(), Python selects the preferred encoding based on the locale that it is running in.
Changed the way multiple cycles are executed in multi-threaded mode to allow tests from different cycles to execute in parallel instead of waiting for each cycle to fully complete before starting the next cycle. This improved parallelism makes it much easier to reproduce race conditions demonstrated by a single testcase, which was not possible with the previous threading behaviour. To maintain existing behaviour for users who have provided a runner.cycleComplete() method, concurrent cycle execution will be disabled if cycleComplete() is overridden. Anybody affected by this is encouraged to transition away from use of cycleComplete() and perform any required cleanup tasks in BaseTest.cleanup() or BaseRunner.testComplete() instead.
Added <requires-python> and <requires-pysys> elements to the project XML file which allow checking for the specified minimum python or pysys version, resulting in a clear error if attempting to use the wrong version.
Added support for coloring console output to highlight passes, fails, warnings and more. This is configured in the project configuration file. Coloring can also be enabled or disabled for a particular user and/or machine using the PYSYS_COLOR=true/false environment variable override. Coloring works on any terminal that supports ANSI escape sequences (e.g. most Unix terminals). On Windows, which does not, it is possible to get colored output by installing a package such as "colorama", which PySys will load if it is present on the python path. It is possible to customize the colors used or to use alternative libraries for coloring on windows by providing a custom ColorLogFormatter class. The colors used for each category of log messages can be customized in the project configuration file, e.g.
<formatter><property name="color:timed out" value="MAGENTA"/></formatter>
Added ProcessUser.getExprFromFile helper method to automate the common task of retrieving some text from a file, for example to capture information such as a process identifier from a log file, or to extract some performance results that were logged.
Added BaseTest.reportPerformanceResult() and a flexible framework for recording performance results (e.g. throughput, latency etc) measured by PySys tests, including storage of results in a human-readable and machine-parsable CSV file together with run-specific information such as the host where the test was executed. The CSV files can be aggregated across multiple test runs and/or cycles and imported into any spreadsheet for comparisons and more detailed analysis. The standard CSVPerformanceReporter can be subclassed and replaced with an alternative recording mechanism if desired (e.g. writing directly to a database or other file format). Fibonacci_test_005 demonstrates how performance results can be reported using this framework.
Added support for providing a custom class to implement formatting of log messages, for both run.log and stdout. Errors in the <formatters> XML node will now be treated as errors rather than being silently ignored.
Changed pysys.py to ignore trailing slash characters on test ids, which makes it easier to use shell tab completion to select a specific test.
Fixed pysys.py command line parser to give a clear error if requested to execute a non-existent test id, or if the test descriptor XML could not be parsed. Previously invalid test ids would be either silently ignored without an error, or would result in other test ids being executed more than once.
Fixed ProcessUser.startProcess to use the test output directory (rather than the current working directory) as the root when a relative path is specified for the workingDir argument.
Fixed bug in which log level and exception tracebacks were being inadvertently suppressed from the stdout console output when executing from multiple threads.
Fixed manual tester thread to report a BLOCKED outcome instead of hanging if a fatal error occurs (e.g. Tck does not load due to DISPLAY not being configured correctly).
Added BaseResultsWriter class and associated docstring documentation to make it easier to create new results writers.
Changed standard record writers to report the number of cycles starting from 1 rather than from 0 (which is consistent with how cycles are displayed by the rest of PySys).
Extended the concept of "writers" to include not just "record" writers (which are enabled only when --record is specified) but also "summary" writers which are always enabled and log a summary at the end of test execution (if none is explicitly configured a default ConsoleSummaryResultsWriter is instantiated), and "progress" writers which are enabled only when --progress is specified and log progress information throughout a run.
The monolithic logic for writing a summary to the console at the end of test execution has been refactored out of baserunner and into the configurable and separately extendable ConsoleSummaryResultsWriter class. Any baserunner subclasses that are currently overriding the summary printing functionality and/or making use of the results dictionary returned by start() should now switch to using "summary" writers instead. This functionality will be removed in a future release and is now deprecated.
The default summary results writer now has a configuration parameter showOutcomeReason which causes the outcome reason string to be included underneath each failure outcome, to provide a quick summary of what went wrong.
The default summary results writer now has a configuration parameter showOutputDir which causes the path to the test's output directory to be printed underneath each failure outcome, to make it easy to quickly find and open the relevant files to debug the failure.
Added a --progress command line option (can also be switched on using the PYSYS_PROGRESS=true environment variable), which logs a summary of how many test have executed, outcomes, a list of most recent failure reasons and a list of what other tests are currently executing. This provides very helpful feedback to the user while executing a long test run. The progress reporting is implemented in a fully extensible way using a new kind of 'progress' result writer. A custom progress result writer class can be configured for a project; if none is specified the default ConsoleProgressResultsWriter is added automatically.
Fixed unexpected DEBUG logging on standard output after any of the Python logging.info/warn/error() methods is called. This behaviour was triggered if certain libraries (e.g SSL libraries) were not available when python starts.
Added defaultIgnoreExitStatus project property which controls whether non-zero return codes from startProcess() result in test failures, when the ignoreExitStatus flag is not explicitly specified. To retain the same behaviour for existing projects, defaultIgnoreExitStatus is set to True if the property is not configured in the project configuration. However to promote best practice for new PySys projects, the example pysys project configuration file sets defaultIgnoreExitStatus to False, which ensures that processes that return failure codes are not ignored unless explicitly intended by the author of the testcase.
Fixed waitForSocket, which in previous versions immediately returned success instead of waiting a valid socket connection as documented.
If the test run is interrupted from the keyboard, the prompt that asks whether to continue to run tests is no longer displayed if there are no more tests left to run. The prompt can also be completely disabled using an environment variable PYSYS_DISABLE_KBRD_INTERRUPT_PROMPT=true, for users who prefer Ctrl+C to immediately terminate the test run in all cases.
Added pysys.utils.pycompat module containing a small set of helpers for writing code that works with Python 2 and Python 3.
Fixed writing to process stdin so that if a character string is passed in it will be converted to a byte object automatically, using the default encoding. Previously, it was not possible to write character strings in Python 3, and in Python 2 it would only work if they contained only ascii characters.
- Added the errorExpr argument to the waitForSignal method. Occurrence of any matches to expressions in this argument will terminate the waitForSignal loop, allowing early exit prior to the timeout.
- Refactored reconfiguration of global logging out of the pysys __init__.py class into the pysys.py launcher. This allows other applications making use of the PySys framework to make their own logging decisions.
- Improved usability of the assertDiff method by writing out the unified diff to a file in the output subdirectory so failures are easier to triage.
- Added the literal argument to the assertGrep method to avoid having to escape regular expressions.
- Added the utils.fileutils module for miscellaneous file related utilities.
- The validateOnly option has been added to the pysys.py run launcher task. When set the purge output subdirectory, setup and execute methods on the test will not be invoked. This makes it easier to fix validation errors in the test without the need to re-run the entire test.
- The logFileContents() method has been added to the pysys.basetest.BaseTest class to allow logging of file contents to the run.log. This can be used to provide additional diagnostic information to the run.log to assist the triage of test failures.
- The CSVResultsWriter has been added to the set of test summary writers. See the pysysproject.xml file in pysys-examples for more details.
- It is now possible to specify a regex for matching in the test selection. See the run usage for more details (pysys.py run -h).
- This release introduces optional fail fast semantics at a macro and micro level. At a macro level this is either through the "defaultAbortOnError" project property, or through the "-b|--abort" option to the pysys launcher run task. See the pysysproject.xml file in pysys-examples, and the run task help usage respectively for more details. At a micro level, all assert and process related methods now take an optional "abortOnError" parameter to override any macro setting. When enabled any error will cause the test to immediately fail, reporting the failure reason.
- Outcomes which are considered a fail now log information as to the cause of the failure. Additionally a call record is reported, giving a comma separated list of "module:lineno" entries detailing the call stack up to the test class instance. This is to aid diagnosing test failure causes.
- The test title is now output to console when running the test.
- The BaseRunner class now contains an isPurgableFile() method. This method can be overridden by any extensions to denote if a zero length file should be purged from the output subdirectory after running of the test.
- It is now possible to register cleanup functions in the BaseTest to negate the need to override the cleanup() action where a call to BaseTest.cleanup(self) must specifically be made. See the epydoc for the addCleanupFunction() in the ProcessUser module.
- Added Darwin as a supported platform.
- Added the maker tag to the pysysproject file to allow specifying a custom test maker class, e.g. to create specific run templates etc. See the pysysproject.xml file in pysys-examples for more information.
- The make option to pysys.py now accepts the testcase directory to be specified to a value other than the current working directory.
- The method getNextAvailableTCPPort has been added to the pysys.basetest.BaseTest class to allow users to allocate server TCP ports in a robust manner.
- The unix and windows process helpers have been updated to fix handle leaks (defect #11 "ProcessMonitor leaks file handles"), and to delete the stdin queue when processes go away.
- Fixed issue with the determination of the overall test outcome due to the incorrect use of the inbuilt sorted() function. The issue meant the test outcome list was not correctly sorted based on precedent, leading to the incorrect determination of the overall test outcome.
- Fixed issue in the pysys.basetest on handling FileNotFoundExceptions in the assert* methods. The exception was not being caught, leading to subsequent asserts in the test class not being performed.
- The PySys framework has been updated to be compliant with conversion to Python 3.x with the 2to3.py conversion script. Installation on Python 3.x is now supported via the source distribution bundle, where the 2to3.py script is run automatically at install time. See details below for installing the source distribution. A binary distribution installer for windows will be included in a later release.
- There are now separate 32 and 64 bit binary distribution installers for windows.
- On failure of the assertLineCount method, the log output now contains the returned number and requested condition (tracker #3045931)
- Each assert method now takes an "assertMessage" parameter to be written to the log output on execution (tracker #3045924). See test PySys_internal_053 in the example testcases for example usage.
- Added the JUnitXMLResultsWriter to log test results in Apache Ant JUnit XML format (one output file per test per cycle). This is useful for integration into Continuous Integration build systems, e.g. TeamCity. The TextResultsWriter and XMLResultsWriter now support the outputDir property to specify the location to write the output files. See the pysys-examples pysysproject.xml file for more details.
- Added the ability to run suites of pyunit tests wrapped up as a single PySys test. This capability is exposed through the PyUnitTest class contained in the pysys.unit.pyunit module. See the pysys-examples pyunit tests for example usage.
- Fix to the unix process helper to correctly set the working directory of child processes in the fork and exec.
- When running tests in parallel, a value of zero given for the -n|--threads option to the run task of the pysys.py launcher, will set the number of threads to the number of available CPUs.
- Updated the pysys.process.plat-win32.helper.ProcessWrapper module to eliminate the use of threads to collect the stdout and stderr from the process via pipes. The module now directly uses win32file.CreateFile to create file objects to pass to the call to win32process.CreateProcess.
- Added the <formatters/> element to the pysysproject file. This allows setting the format of the test output to stdout and the runlog in accordance to the format specifiers in the python logging and time modules. For examples of the use of this element, see the pysysproject file included in the PySys examples.
- Logging of exceptions and failed asserts has been changed from info to warn level (tracker #2784251).
- Added extra debug logging in pysys.utils.filegrep, and pysys.basetest for when performing asserts against a line count in an input file (tracker #2824758).
- The testcase output summary is now printed on termination of the test run via a keyboard interrupt (tracker #2816212).
- The PySys project file now allows assignment of the project root location to a variable which can then be used for later expansion within the file. This allows the definition of project variables to include the full path where this is required, e.g. XSL stylesheets which must use the full path to the file rather than a relative path etc. Note that modules within PySys can reference the project root location directly using PROJECT.root (tracker #2795316).
- The pysys.baserunner class now passes the -X arguments into the test summary writer setup action to allow logging of the user supplied extra arguments(tracker #2814499). The pysys-log.xsl stylesheet used by the XMLResultsWriter has been updated to display this information in the test summary display.
- Fixed an issue where when the pysysproject file was missing, defaults for the runner module and the test output summary writer were not being set.
- Fixed a defect in the unix process helper module to correct a file handle leak in the write end of the stdin pipe.
Added the ability to run tests concurrently through the -n | --threads option to the pysys launcher run target. Tests to be run are placed on a request queue and processed by the designated number of worker threads. The results of each test are then placed on a result queue, collated and displayed in the order in which they would run serially. Depending on the nature of the application under test, the recommended number of threads to designate when using this option is no more than two times the number of CPUs. Note also that care needs to be made when running tests in parallel, so as to ensure no shared resources are accessed in a non-atomic way, e.g using direct references to os.environ() in one test when another test modifies the environment directly etc.
The constructor to the pysys.baserunner.BaseRunner class was changed to include the threads parameter, i.e.
def __init__(self, record, purge, cycle, mode, threads, outsubdir, descriptors, xargs)
This parameter is required for the runner to create the required threadpool before running a set of tests in parallel. Any custom runner classes extending the base runner will need to be updated to incorporate this change.
Removed module specific loggers from pysys in order to support running tests in parallel. There is now a single logger used within the framework, and which can be referenced directly from the pysys.constants module. Attached to this logger are two handler types; one for logging to stdout, and one for logging to the run log file saved in the output subdirectory of each test. The stdout handler is set to only log to stdout from the main thread, whilst the run log file handlers are set to log to the output subdirectory of a test only on the creating thread.
Added exception handling to the pysys.process.user module when trying to stop all processes on destruction. When a background process takes longer to stop than the default timeout period, the thrown pysys.exceptions.ProcessTimeout exception was uncaught causing abnormal exit from the test run.
- The clean target has been updated to accept the -a | --all command line option to allow deleting all derived files produced when running a set of testcases, i.e. both the testcase output subdirectory and any compiled test class modules.
- The waitForSignal method of the ProcessUser class, subclassed by both the BaseTestand BaseRunner classes, has been updated to return a list of match objects on invocation. By using tagged regular expressions in the expr parameter of the method call, this allows retrieval of portions of the matched data e.g. to extract expressions in the file to use later in the validation routines.
- All references to pysys.constants.TRUE and pysys.constants.FALSE have been replaced by the native Python True and False literals. The values of the constants have been set to True and False respectively so as to maintain backwards compatibility.
- The PySys test and PySys project files have been renamed by default from .pysystest to pysystest.xm, and .pysysproject to pysysproject.xml respectively. Backwards compatibility is maintained for the previous file naming convention, though this will be deprecated in a later release; it is strongly advised that the new naming convention is adopted. New tests made using the PySys launcher will by default use the new naming convention. This change was made due to issues on Windows systems displaying and recognising hidden files, and files without specified extensions e.g. within the Eclipse framework, for display in internet browsers etc.
- The clean mode of operation has been added to the pysys.py launcher. This allows removal of testcase output subdirectories, e.g. before importing into a source code control system. The -o option allows specifying the output subdirectory name to be deleted, which defaults to the platform identifier if not specified.
- The test output summary writer interface has been changed so that the test output is written and updated during the test execution; previously a call to the writer was only made on completion of the test run. This allows monitoring the test output summary during the test execution to monitor the run time status of the tests.
- Added the XMLFileResultsWriter class to the pysys.writer module. This performs logging of the test output summary in an XML format, suitable for display via XLST in a web browser. A simple XSL stylesheet is included with the PySys distribution to provide better display in internet browsers.
- Added the ability to specify custom test output summary writers in the PySys project file via the <writer> tag. For an example see the .pysysproject file in the pysys-examples distribution. Should no <writer> be specified in the project file, the default XMLFileResultsWriter will be used. Multiple writers may be specified in the PySys project file.
- Added exception logging on parsing errors in the PySys project file, e.g. when the file in badly formed due to invalid XML tokens etc.
- Added variable argument passing to the process.monitor.ProcessMonitor class constructor so that operating specific arguments can be passed into the class on instantiation. The wrapper method pysys.basetest.BaseTest.startProcessMonitor has also been updated to allow pass through of the variable arguments.
- The win32 process.monitor module has been changed so that on windows systems the percentage CPU usage is not normalised by default by the number of available processors, e.g. on a 4 core processor if 2 cores were fully utilized the CPU usage was previously output as 50% - the change means that the reported usage will now be 200% (a value of 100% indicates that one core is fully utilitised). This makes the output consistent with that reported on unix systems. Should the previous behavior be required the numProcessors argument can be passed to the pysys.basetest.BaseTest.startProcessMonitor method in order to normalise the CPU usage statistics by the number of processors. On windows systems the number of processors can be obtained from the NUM_PROCESSORS environment variable.
- Added comments to the PySys Project file distributed with the example testcases, to detail the possible configuration options.
- The lastgrep method has been added to pysys.utils.filegrep, and the assertLastGrep method has been added to the BaseTest class. This allows test validation to be performed based on regular expression matching on the last line of an input file to the assertLastGrep method.
- The win32 process monitor has been modified to calculate the percentage CPU usage statistics as a sum over all available processors. A CPU usage of 100% represents the process fully utilising all available processors.
- The win32 process monitor now also logs the handle count of a process.
- Fixed a bug in pysys.process.user.ProcessUser destructor to explicitly set the process list to null to allow process handles to be cleaned up on destruction. This bug only seemed to be exhibited when the process handle of a process returned in the startProcess() method was set as a data attribute to an instance of the class. This handle was then both a data attribute of the class, and was contained in a list data attribute of the class. Under these conditions the handles were not being released correctly.
- The print mode of the pysys.py launcher now supports printing out the test user defined modes, and the printing out of tests that can be run in a given mode.
The OSFAMILY constant has been added to pysys.constants, and takes the value 'windows' on all win32 operating systems, and 'unix' on sunos and linux operating systems. The value of the OSFAMILY can be used within the .pysysproject file using the osfamily attribute to the <property> element. This allows capturing the value to be used in expansion of other properties defined within the project file; see below for an example usage. Should no value be set in a properties file, a default value of "osfamily" is assumed.
The .pysysproject file now allows explicitly setting the environment value to be used in expansions via the environment attribute to the <property> element; see below for an example usage. Should no value be set in a properties file, a default value of "env" is assumed (this allows for backwards compatibility).
The .pysysproject file now takes the file attribute to the <property> element. This allows properties to be read from file, where the properties are specified in the name=value syntax, e.g.
<pysysproject> <property environment="env"/> <property osfamily="osfamily"/> <property file="${osfamily}.properties" /> <property name="lib" value="${library}_${osfamily}_${version}_${env.USER}.so"/> </pysysproject>where the property file contains the following:
version=1.0 library=jstore${version}.jarFor more details, see testcase PySys_internal_002 in the pysys-examples/internal area which demonstrates this.
Fixed the issue of removing zero size files from the output subdirectory on win32 platforms; was due to the stderr and stdout file handles not being properly closed down. Updated the BaseRunner to attempt to remove the zero sized files 3 times to try to avoid race conditions of stopped background processes holding on to the file handles too long before dying.
The win32 process helper now ensures the environment in which the process runs is converted to unicode to avoid issues encountered with running under certain locales.
- The pysys.process.ProcessUser class has been added to define an interface to subclasses which use the underlying process helper classes. Both the BaseTest and BaseRunner classes now extend this so as to provide a common interface for process manipulation. A common paradigm for creating extension modules to PySys is to create a helper class which provides the methods for starting an interacting with the application under test (AUT). These helper classes have a call back to an instance of the ProcessUser so that it can make use of the high level process methods. As both the BaseTest and BaseRunner classes are instances of the ProcessUser, the extension module helper classes can be used in extensions to both of these to allow the AUT to be started both within a testcase, and within the runner.
- The method signature to the pysys.utils.filereplace replace method has been changed to set the default value for the marker to the empty string
- Bugs fixes for cleaning up leakage of threads from the process helpers, and file handle leakage from the base runner classes.
- Fixed a bug a testcase was not being marked as BLOCKED when unable to start a process using the process helper module.
- Failure on the assertOrderedGrep now prints out the line the failure occurred on.
- Fixed a bug where timedout processes started in the foreground were not being stopped automatically at the end of the testcase.
- The default name of the PySys test descriptor file has been changed from "descriptor.xml", to ".pysystest". This change is to maintain a consistent naming convention across configuration files within the framework, e.g. the project file ".pysysproject" denotes the project root and project specific information, whilst a test file ".pysystest" denotes a testcase within the project, and contains meta data for the test. Support for the previous name is maintained, though it should be noted that testcases created with the 'pysys.py make' command will have the new naming convention used.
- The windows installer has been updated to add shortcuts to the uninstaller, and to create a separate directory for the inclusion of project extensions.
- The getInstanceCount method has been added to the pysys.basetest.BaseTest class to reference count the number of named processes started during a test run. The startProcess method of the class adds a reference count to an internal dictionary structure keyed on the displayName passed into the method to achieve this.
- The writeProcess method has been added to the pysys.basetest.BaseTest class to provide a wrapper around the write method of the underlying process helper class. This wrapper perform a check on the running status of the process prior to the write, and performs additional logging to the run.log to audit the write.
- Fixed a bug in the replace method of the filereplace module, where the method signature was missing the marker parameter
- Added support to the pysys project file to allow adding path locations to the Python path. See the .pysysproject file in pysys-examples for more detail.
- Release was superseded immediately by the 0.3.3 release. See release notes for new features for 0.3.3 for more information.
- The process helper modules have been updated to allow the writing to the stdin of a process via the write() method on the process handle.
- Several bug fixes have been applied to the unix process helper module.
- The pysys-examples/internal directory has been added to the examples module. This will contain internal testcases for self testing the pysys framework. These have been included in the distribution as examples of the use of the framework.
- The pysys project file has been added into the framework to allow the setting of project specific constants within the application. The project file should be written to the base location of the project, with the filename .pysysproject. The location of this file denotes the root location of the project. For an example of the file see pysys-examples/.pysysproject. Any name value properties within the file will be set as data attributes of the pysys.Project class, which is referenced in the pysys.constants module using the variable PROJECT.
- Updates to the epydoc output for documenting the classes and modules of pysys
- Addition of the pysys.py module for printing, running and making new testcase directory structures. This allows a single distributed script to be used to perform all functionality available from the console.
- Remove of the run method from the console launch helper.
- The suites element in the test descriptor has been renamed to groups. This is to allow testcases in a single directory to be classified as a single testsuite, and subsets thereof to be regarded as a group
- Minor bug fixes to the manual tester and process module
- The Manual Tester has been updated to support the <expectedresult> element in the input xml file. This allows display of the expected result for a manual step to be presented in the user interface. The ability to optionally record a defect in the log output is also now included.
- The createDescriptors method has been removed from the pysys.launcher.console package and moved into pysys.launcher. This allows the utility method to be used for other custom launchers.
- This release includes updates to the Python doc strings for automated generation of epydoc using the Epydoc package (http://epydoc.sourceforge.net). The installer now distributes the generated epydoc in the site-packages/pysys-doc directory. For windows installs a link to the epydoc and release notes is now added as a link in the start menu items
- Added the setup() method to the BaseTest class to allow custom setup actions to be performed prior to execution of a particular test case
- Fixed a bug where if the --type option was not supplied to ConsoleMakeTestHelper as a command line option, the resulting descriptor had type="None"
- The Manual Tester UI has been updated so that it can be resized, and is easier to navigate through the tests.
- The BaseRunner start method now takes an optional list of result writer class instances to perform test audit logging at the end of a test cycle. This allows custom result writers to be passed to the runner to, for example, write the results to a database, proprietary system etc
- Added the ability to differentiate between automated and manual testcases using the test attribute to the pysystest element in the testcase descriptor. If the attribute is not present the test will be assumed to be an automated test. The runTest and printTest launch helpers allow you to differentiate between automated and manual tests using the --type command line argument. For more information see the examples in pysys-examples
- Added support for the requirements traceability. This includes printing requirements covered by a set of testcases, and running testcases which cover a particular requirement id
- Added the ConsoleMakeTestHelper class to pysys.launcher.console. This facilitates the creation of new testcase structures from the command line. Updated pysys-examples/fibonacci to demonstrate the use of the utility class