Added Xfail and Mayfail attributes to city runner #686

coldav · 2025-02-19T17:18:41Z

Overview

Support attribute Xfail and Mayfail for city runner and an add a --override option to override entries in the main csv file.

Reason for change

We want to be able to run tests that we know fail, but we don't want an overall fail.

Description of change

Added xfail and Mayfail as attributes to city runner.

Xfail means we expect it to fail and will give a failing return code if it unexpectedly pass, otherwise it does not contribute to an overall fail.

Mayfail means it sometimes fails and will not contribute to a failing return code.

Unlike some of the other attributes, these cannot be set with additional csv files, as it was felt this leads to overcomplexity. Instead it is expected that some form of csv combination script could be used alongside this.

Added --override option. This provides an additional csv file which can override entries in the main csv file if it matches the first two values in an entry. This is useful for overriding attributes such as Xfail.

Anything else we should know?

The following is example output

dump.txt

Checklist

Read and follow the project Code of Conduct.
Make sure the project builds successfully with your changes.
Run relevant testing locally to avoid regressions.
Run clang-format-19 on all modified code.

.github/actions/run_opencl_cts/action.yml

scripts/testing/city_runner/test_info.py

scripts/testing/city_runner/runner.py

…ntially There was a race condition in updating the cache as it just ran the tests overnight in parallel. Move all the build parts of PR jobs to an action. Add a workflow pr_tests_cache.yml to seed the caches by calling the actions in sequence. The scheduled run_pr_tests overnight will no longer have the ability to seed the cache and this will only be done via pr_tests_cache.yml. This does not yet delete the caches. We can't be sure this works so this is initially commented out.

Support attribute Xfail and Mayfail for city runner. These can be set as additional elements in the csv file. Xfail means we expect it to fail and will give a failing return code if it unexpectedly pass, otherwise it does not contribute to an overall fail. Mayfail means it sometimes fails and will not contribute to a failing return code. Unlike some of the other attributes, these cannot be set with additional csv files, as it was felt this leads to overcomplexity. Instead it is expected that some form of csv combination script could be used alongside this.

coldav requested a review from a team as a code owner February 21, 2025 10:07

coldav force-pushed the colin/add_xfail_city_runner branch from 8142c1d to be7fd1f Compare February 21, 2025 10:15