Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add differential testing harness #216

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

LegNeato
Copy link
Collaborator

@LegNeato LegNeato commented Feb 4, 2025

The test harness compiles and runs Rust
shaders/kernels and compares the output with those produced by equivalent WGSL
shaders/kernels. Unlike traditional reference, golden, or snapshot tests—which check
against static outputs—differential testing compares multiple independent implementations of the same logic.
This approach is more robust because discrepancies between them are more likely to
indicate real bugs rather than issues with outdated reference files.

You can run difftests via cargo difftest. This is an alias set up in .cargo/config
for cargo run --release -p difftest --. You can filter to run specific tests by
passing the (partial) filenames to cargo difftest some_file_name.

@LegNeato LegNeato marked this pull request as ready for review February 6, 2025 23:29
@LegNeato

This comment was marked as outdated.

@LegNeato
Copy link
Collaborator Author

Perhaps better DX is to have directories and just cargo run --release in each?

@LegNeato
Copy link
Collaborator Author

@Firestar99 FYI, I am going to revamp this a bit as it is a bit of a mess.

@LegNeato LegNeato marked this pull request as draft March 13, 2025 15:37
@LegNeato
Copy link
Collaborator Author

FWIW, I am almost ready to update this. It is much cleaner now :-)

@LegNeato LegNeato force-pushed the difftests branch 2 times, most recently from 2b484a8 to 4408a6e Compare March 17, 2025 01:17
@LegNeato LegNeato marked this pull request as ready for review March 17, 2025 01:21
@LegNeato
Copy link
Collaborator Author

LegNeato commented Mar 17, 2025

I think this is ready for review. Sorry it is so big! I can break it down if need be.

The main changes:

  1. Move compiletests under tests/ and add difftests there as well.
  2. Add a difftest harness
  3. Add a difftest sdk/library used by the binaries under test.
  4. Add a simple compute difftest test using wgpu (one variant written in rust, one in wgsl).
  5. Add to CI (on linux install software GPU so the tests will run)
  6. Some docs in a README.

@LegNeato
Copy link
Collaborator Author

This is what a run with passing and failing tests looks like:

Screenshot 2025-03-16 at 8 15 09 PM

On failing tests, the outputs are grouped as you may have 4 variants where 3 are the same and 1 is different. In that case the 3 same will be grouped in the output so it is easier to understand what is going on.

This runs wgsl shaders and rust shaders and compares the output.
If the output differs, the test fails. Differential testing is better
than snapshot testing or golden file testing as there are no reference
files to get outdated.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant