Skip to content

Commit

Permalink
Support system tests with arbitrary many *.lobster files
Browse files Browse the repository at this point in the history
Add `README.md` to explain how system tests work.
  • Loading branch information
phiwuu committed Dec 17, 2024
1 parent eb9dade commit 69c835a
Show file tree
Hide file tree
Showing 2 changed files with 42 additions and 3 deletions.
41 changes: 41 additions & 0 deletions tests-system/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
The script `run_tool_tests.py` executes system tests.
It can execute system tests for all lobster tools.

The one and only command line argument to the script must be the name of the tool under
test.
For example, to run system tests for `lobster-trlc`, the tool must be started like this:
```
> python3 run_tool_tests.py lobster-trlc
```

System test cases always consist of the following folder structure:
```
test-case/
├─ input/
│ ├─ args.txt
expected-output/
├─ exit-code.txt
├─ stderr.txt
├─ stdout.txt
```

The target `make system-tests` in the root `~Makefile` sets the current working
directory of the tool under test to the `input` folder.

The files must contain the following piece of information:
- `args.txt`:
This file shall contain the command line arguments that must be used to start the tool
under test. Each line represents a separate argument.
- `exit-code.txt`: This file shall contain the expected exit code of the tool under test.
- `stderr.txt`: This file shall contain the expected standard error stream of the tool
under test.
- `stdout.txt`: This file shall contain the expected standard output stream of the tool
under test.

Any additional files needed by the tool under test must be located in the `input`
folder.

The expected values will all be compared against their actual values, and test cases
only count as "passed" if the values match.

4 changes: 1 addition & 3 deletions tests-system/run_tool_tests.py
Original file line number Diff line number Diff line change
Expand Up @@ -51,9 +51,7 @@ def _get_expected_lobster_output_file_name(self) -> str:
Note: The system test must always be prepared such that the tool under test
generates the lobster file in the "input" folder. Other test setups are not
supported. Furthermore, the tool under test must generate exactly one output
file. Tools like 'lobster-cpptest' are able to generate multiple files. Testing
that is currently not supported.
supported.
"""
for dir_entry in scandir(self.get_expected_output_path()):
if (not dir_entry.is_dir()) and dir_entry.name.endswith(".lobster"):
Expand Down

0 comments on commit 69c835a

Please sign in to comment.