You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We currently run 5 repeats of test jobs per configuration in order to get some information about test stability. This is quite inefficient. We should consider two options:
Stop running repeats if they don't often find stability issues. This can be observed by looking at per-PR metadata-update commits that mark a test as intermittent. In this case we'd rely on the final landing job to find intermittent tests, which might be worse (because in that case we end up having to rerun all the tests in a chunk, not just the changed tests).
Run the tests on repeat in a single job. This involves a number of complications:
We need a way to configure a try run with this setting. In terms of wptrunner we could use the --repeat command line argument, but I don't think there's an existing mechanism to pass that in via try syntax.
We need to ensure we end up with a usable wptreport.json file that works with the metadata update.
For PRs that affect many tests, the jobs might end up running too long if we have multiple repeats in a single job. We either need to find a way to time-limit at the wptrunner level (i.e. stop the repeats if we hit some time threshold) or otherwise when we hit some threshold of tests switch back to one repeat per job and multiple jobs per configuration.
The text was updated successfully, but these errors were encountered:
jgraham
changed the title
Consider running downstry trypushes with one job per configuration
Consider running downstream trypushes with one job per configuration
Jul 4, 2022
We currently run 5 repeats of test jobs per configuration in order to get some information about test stability. This is quite inefficient. We should consider two options:
--repeat
command line argument, but I don't think there's an existing mechanism to pass that in via try syntax.The text was updated successfully, but these errors were encountered: