Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consider running downstream trypushes with one job per configuration #1455

Open
jgraham opened this issue Jul 4, 2022 · 0 comments
Open

Comments

@jgraham
Copy link
Member

jgraham commented Jul 4, 2022

We currently run 5 repeats of test jobs per configuration in order to get some information about test stability. This is quite inefficient. We should consider two options:

  • Stop running repeats if they don't often find stability issues. This can be observed by looking at per-PR metadata-update commits that mark a test as intermittent. In this case we'd rely on the final landing job to find intermittent tests, which might be worse (because in that case we end up having to rerun all the tests in a chunk, not just the changed tests).
  • Run the tests on repeat in a single job. This involves a number of complications:
    • We need a way to configure a try run with this setting. In terms of wptrunner we could use the --repeat command line argument, but I don't think there's an existing mechanism to pass that in via try syntax.
    • We need to ensure we end up with a usable wptreport.json file that works with the metadata update.
    • For PRs that affect many tests, the jobs might end up running too long if we have multiple repeats in a single job. We either need to find a way to time-limit at the wptrunner level (i.e. stop the repeats if we hit some time threshold) or otherwise when we hit some threshold of tests switch back to one repeat per job and multiple jobs per configuration.
@jgraham jgraham changed the title Consider running downstry trypushes with one job per configuration Consider running downstream trypushes with one job per configuration Jul 4, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant