Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TST: dask tests are extremely flaky #263

Merged
merged 4 commits into from
Mar 1, 2025
Merged

Conversation

crusaderky
Copy link
Contributor

@crusaderky crusaderky commented Feb 28, 2025

Dask is running on 5 cases per test, causing it to be extremely flaky.
This PR brings it back in line with other libraries to 200.
This PR tweaks xfails and skips after running locally on 2000 examples and adds some commentary.

@ev-br
Copy link
Member

ev-br commented Feb 28, 2025

All other test runs are under five minutes, and even with max-examples 5, dask tests are the slowest. I'd rather strongly prefer all tests here to run under a few minutes.

Even the default 200 examples is more of a smoke-testing really. When looking for regressions or working on a new feature, I regularly run with --max_examples=10_000 locally. Not on dask, of course, that would never complete.


The state of the CI currently is not great indeed. One possibility is to have a nightly job which runs many examples.
One other idea is to export arrays in "unvectorized" tests to dlpack or something: data-apis/array-api-tests#197 (comment).

@crusaderky
Copy link
Contributor Author

This is very frustrating. I've reverted the number of examples back to 5 and ran locally with 2000 examples. The updated xfails and skips files reflect that.

@ev-br ev-br merged commit 158caf2 into data-apis:main Mar 1, 2025
40 checks passed
@ev-br
Copy link
Member

ev-br commented Mar 1, 2025

Thanks @crusaderky !
It is frustrating, yes. Suggestions for improvements are most welcome indeed

@ev-br ev-br added this to the 1.12 milestone Mar 1, 2025
@crusaderky crusaderky deleted the dask_xfail branch March 4, 2025 17:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants