Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci: Fail CI if xfails pass unexpectedly #1888

Merged
merged 3 commits into from
Feb 26, 2025

Conversation

romanc
Copy link
Contributor

@romanc romanc commented Feb 26, 2025

Description

Be strict about tests marked with xfail. If they pass unexpectedly, fail the testsuite instead of logging an XPASS.

Requirements

  • All fixes and/or new features come with corresponding tests.
  • Important design decisions have been documented in the appropriate ADR inside the docs/development/ADRs/ folder.
    N/A

Be strict about tests marked with `xfail`. If they pass unexpectedly,
fail the testsuite instead of loggin `XPASS`.
@romanc
Copy link
Contributor Author

romanc commented Feb 26, 2025

@tehrengruber it looks like

@pytest.mark.uses_tuple_args
@pytest.mark.xfail(
reason="Not implemented in frontend (implicit size arg handling needs to be adopted) and GTIR embedded backend."
)
def test_tuple_arg_with_different_but_promotable_dims(cartesian_case):
@gtx.field_operator
def testee(a: tuple[cases.IField, cases.IJField]) -> cases.IJField:
return a[0] + 2 * a[1]
cases.verify_with_default_data(
cartesian_case,
testee,
ref=lambda a: a[0][:, np.newaxis] + 2 * a[1],
)

passes with the numpy backend despite the general xfail. We could add strict=False to this test case. What do you think?

@havogt havogt marked this pull request as ready for review February 26, 2025 13:04
@havogt havogt requested review from tehrengruber and egparedes and removed request for tehrengruber February 26, 2025 13:04
Copy link
Contributor

@egparedes egparedes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@@ -93,6 +93,7 @@ def _process_args(
el_range == range_ and el_dims == dims
for (el_range, el_dims) in ranges_and_dims
):
print(ranges_and_dims)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Debugging leftover?

@romanc
Copy link
Contributor Author

romanc commented Feb 26, 2025

Nice! Thanks for fixing this 🚀

@havogt havogt merged commit 3ba7e62 into GridTools:main Feb 26, 2025
31 checks passed
@romanc romanc deleted the romanc/xfail-strict branch February 27, 2025 06:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants