Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
feat(ct): add
flaky
test case property
Albeit there's already the `{repeat_until_ok, N}` TC property, it's arguably of reduced use in a CI environment, as any failure will fail the whole suite, even if the final execution succeeds. Here we introduce a new kind of TC property: `{flaky, N}`, where `N` is a positive integer. It works similarly to `{repeat_until_ok, N}`: the TC is repeated up to `N` times until it succeeds or retries are exhausted. The TC gets the status of its last run: if it eventually succeeded, it's considered a success. Example usage: ```erlang -module(my_SUITE). all() -> %% This test case will be run up to 10 times. [{testcase, t_my_flaky_test, [{flaky, 10}]}]. t_my_flaky_test(Config) -> K = {?MODULE, ?FUNCTION_NAME}, N = persistent_term:get(K, 0), case N > 5 of true -> ok; false -> persistent_term:put(K, N + 1), error(boom) end. ``` Execution logs: ``` Testing lib.my_SUITE: Stopping test case repeat operation: {flaky,10} Testing lib.my_SUITE: TEST COMPLETE, 1 ok, 0 failed of 1 test cases ``` Note that, even though it actually ran 6 times, the final statistics are not duplicated. The produced HTML logs do contain logs from all failures, nevertheless, for debugging purposes.
- Loading branch information