Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Swap test order #3163

Draft
wants to merge 3 commits into
base: develop
Choose a base branch
from
Draft

Swap test order #3163

wants to merge 3 commits into from

Conversation

gonuke
Copy link
Contributor

@gonuke gonuke commented Oct 5, 2024

Description

Change the directory name of python unit tests so that they run before regression tests. This assumes that the test order is determined by the alphabetical order of the test directories.

Fixes #2828

Checklist

  • I have performed a self-review of my own code
  • I have run clang-format (version 15) on any C++ source files (if applicable)
  • I have followed the style guidelines for Python source files (if applicable)
  • I have made corresponding changes to the documentation (if applicable)
  • I have added tests that prove my fix is effective or that my feature works (if applicable)

@gonuke
Copy link
Contributor Author

gonuke commented Oct 5, 2024

This definitely results in the unit tests being run first, which is certainly helpful in getting earlier failures.

For reasons I don't fully understand, changing the order of the regression tests causes universe numbers to be off by about 6390. I am guessing that the test harness is somehow generating unique universe IDs across many test problems, and the unit tests consume roughly 6390 of these IDs before the regression tests begin.

There is a mode to update the reference data, but I'm not sure I understand all of this enough to be sure it's the right step at this time.

Perhaps @paulromano could offer some insight and/or chime in on whether this is worth pursuing.

@gonuke
Copy link
Contributor Author

gonuke commented Oct 5, 2024

Note: While this PR shows 117 files that have changed, most of them are just directory name changes. Four (4) files have been updated because they have python imports that refer to that directory name.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Run Unit Tests First in CI
1 participant