CI: Run some tests with compute-sanitizer #566

carterbox · 2025-04-22T18:06:39Z

Description

Runs python 3.12 pytests in the context of compute-sanitizer to check for memory issues and errors from the CUDA API.

closes #565
closes #562

Checklist

New or existing tests cover these changes.
The documentation is up to date with these changes.

copy-pr-bot · 2025-04-22T18:06:43Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

leofang · 2025-04-22T19:24:56Z

FYI right now we use the mini-CTK approach in the CI:

local CTK: we only grab what we need with a custom GHA

cuda-python/.github/actions/fetch_ctk/action.yml

Lines 85 to 92 in d425a88

    
                   populate_cuda_path cuda_nvcc 
        
                   populate_cuda_path cuda_cudart 
        
                   populate_cuda_path cuda_nvrtc 
        
                   populate_cuda_path cuda_profiler_api 
        
                   populate_cuda_path cuda_cccl 
        
                   if [[ "$(cut -d '.' -f 1 <<< ${{ inputs.cuda-version }})" -ge 12 ]]; then 
        
                     populate_cuda_path libnvjitlink 
        
                   fi

wheel: same is done via the optional dependencies

cuda-python/.github/workflows/test-wheel-linux.yml

Lines 218 to 222 in d425a88

    
                       if [[ "${{ inputs.local-ctk }}" == 1 ]]; then 
        
                         pip install *.whl 
        
                       else 
        
                         pip install $(ls *.whl)[all] 
        
                       fi

conda: not yet implemented (CI: Test conda-based workflows #280) but should be straightforward as you know

So compute-sanitizer is currently not available in the CI. But I assume it can be grabbed easily.

@cryos is refactoring our CI (#555). I suggest we perhaps add another standalone pipeline for running compute sanitizer?

cryos · 2025-04-22T19:58:28Z

I was going to look at tools like this next, that is a great point and something I can factor in. Looking at the proposal here picking out a test run would be reasonable, I know there are other tools we would like to run too.

carterbox · 2025-04-22T22:16:15Z

/ok to test f27e1f4

leofang · 2025-04-22T22:25:46Z

I doubt commit f27e1f4 would work -- we'll see: #571.

carterbox · 2025-04-23T17:34:24Z

/ok to test 05a7068

carterbox · 2025-04-23T21:15:16Z

/ok to test dde857b

carterbox · 2025-04-23T21:57:10Z

Yay! The linux-64 tests are failing for the correct reason! (the compute sanitizer returns non-zero because it has detected issues).

https://github.com/NVIDIA/cuda-python/actions/runs/14628267586/job/41045781037?pr=566

Windows tests are failing because I have disabled them partially.

carterbox · 2025-04-24T18:26:05Z

/ok to test a1ea51e

There is no compute-sanitizer wheel, so we can only run when the ctk is installed system-wide

Because the sanitizer commands depend on the version of the sanitizer we need to be able to run the sanitzer to set the sanitizer cmd. Thus, we need to setup the sanitzer after it is installed.

carterbox · 2025-04-24T20:28:24Z

/ok to test 0430930

carterbox · 2025-04-25T20:27:12Z

/ok to test 9c25910

carterbox · 2025-04-25T21:47:34Z

/ok to test 7fb013e

copy-pr-bot · 2025-04-25T22:48:23Z

Auto-sync is disabled for ready for review pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

leofang · 2025-04-26T03:29:26Z

Let's get #562 (comment) addressed and merge!

cuda_bindings/docs/source/environment_variables.md

.github/workflows/test-wheel-linux.yml

cuda_bindings/tests/test_cuda.py

…sanitizer-to-ci

carterbox · 2025-04-28T16:30:09Z

/ok to test 675c41f

carterbox · 2025-04-28T17:42:38Z

/ok to test 5abd6e3

rwgk

Awesome, I'm really happy that we'll be routinely testing with the compute sanitizer.

.github/workflows/test-wheel-linux.yml

carterbox · 2025-04-28T20:59:02Z

/ok to test 53a01cb

rwgk

Thanks @carterbox!

leofang

Very neat, thanks!

@cryos are you OK if we merge this now, or if you'd rather wait until we merge the CI refactoring (#555)?

kkraus14 · 2025-04-29T14:46:08Z

@cryos are you OK if we merge this now, or if you'd rather wait until we merge the CI refactoring (#555)?

Marcus is on PTO until May 5 so lets merge this and we can sync with him once he's back 😄

leofang · 2025-04-29T16:27:08Z

Right he told me about it but I forgot... let's merge now.

github-actions · 2025-04-29T16:46:00Z

Doc Preview CI
Preview removed because the pull request was closed or merged.

leofang requested review from leofang and cryos April 22, 2025 19:25

leofang assigned carterbox Apr 22, 2025

leofang added P1 Medium priority - Should do CI/CD CI/CD infrastructure labels Apr 22, 2025

This comment has been minimized.

Sign in to view

cryos linked an issue Apr 23, 2025 that may be closed by this pull request

CI: Refactor the CI matrix #576

Open

cryos removed a link to an issue Apr 23, 2025

CI: Refactor the CI matrix #576

Open

carterbox added 10 commits April 24, 2025 15:25

CI: Add compute-sanitizer paths to linux test environment

843130f

CI: Run pytest in the context of compute-sanitizer

c8df0dc

CI: Only use compute-sanitizer for tests of one python version

4ac79d0

CI: Add non-zero exitcode to compute-sanitizer

be912c9

CI: Only run compute-sanitzer when testing against local ctk

865eeb5

There is no compute-sanitizer wheel, so we can only run when the ctk is installed system-wide

CI: Delay CUDA_HOME variable expansion

ddd0714

CI: Move compute-sanitzer setup into own step after CTK setup

5079605

Because the sanitizer commands depend on the version of the sanitizer we need to be able to run the sanitzer to set the sanitizer cmd. Thus, we need to setup the sanitzer after it is installed.

CI: Optionally skip tests that raise CUDA API errors

4a03465

CI: Add sanitizer skip environment variable to CI

bd88039

DOC: Fix spelling of CI step name

0430930

carterbox force-pushed the dching/add-compute-sanitizer-to-ci branch from a1ea51e to 0430930 Compare April 24, 2025 20:26

leofang added the cuda.bindings Everything related to the cuda.bindings module label Apr 25, 2025

BUG: Correctly spell "version"

7fb013e

carterbox marked this pull request as ready for review April 25, 2025 22:48

carterbox requested a review from leofang April 25, 2025 22:48

leofang reviewed Apr 26, 2025

View reviewed changes

cuda_bindings/docs/source/environment_variables.md Outdated Show resolved Hide resolved

.github/workflows/test-wheel-linux.yml Outdated Show resolved Hide resolved

rwgk reviewed Apr 26, 2025

View reviewed changes

.github/workflows/test-wheel-linux.yml Outdated Show resolved Hide resolved

cuda_bindings/tests/test_cuda.py Outdated Show resolved Hide resolved

carterbox added 5 commits April 28, 2025 10:43

DOC: Fix spelling of sanitizer

2632bbc

TST: Define new test skip in conftest instead of copy-paste

2bfd402

Merge remote-tracking branch 'upstream/main' into dching/add-compute-…

3f7a79b

…sanitizer-to-ci

TST: Cleanup post-merge

02559a3

CI: Remove COMPUTE_SANITIZER_VERSION variable

675c41f

carterbox requested review from leofang and rwgk April 28, 2025 16:30

TST: Skip another post-merge error

5abd6e3

rwgk previously approved these changes Apr 28, 2025

View reviewed changes

.github/workflows/test-wheel-linux.yml Outdated Show resolved Hide resolved

TST: Use consistent name for environment variable and test skip function

53a01cb

carterbox dismissed rwgk’s stale review via 53a01cb April 28, 2025 20:58

rwgk approved these changes Apr 28, 2025

View reviewed changes

leofang approved these changes Apr 29, 2025

View reviewed changes

leofang merged commit 034ffbf into NVIDIA:main Apr 29, 2025
75 checks passed

leofang mentioned this pull request Apr 29, 2025

Doc: mention about compute-sanitizer errors #589

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CI: Run some tests with compute-sanitizer #566

CI: Run some tests with compute-sanitizer #566

carterbox commented Apr 22, 2025 •

edited

Loading

copy-pr-bot bot commented Apr 22, 2025

leofang commented Apr 22, 2025

cryos commented Apr 22, 2025

carterbox commented Apr 22, 2025

leofang commented Apr 22, 2025

This comment has been minimized.

carterbox commented Apr 23, 2025

carterbox commented Apr 23, 2025

carterbox commented Apr 23, 2025 •

edited

Loading

carterbox commented Apr 24, 2025

carterbox commented Apr 24, 2025

carterbox commented Apr 25, 2025

carterbox commented Apr 25, 2025

copy-pr-bot bot commented Apr 25, 2025

leofang commented Apr 26, 2025

carterbox commented Apr 28, 2025

carterbox commented Apr 28, 2025

rwgk left a comment

carterbox commented Apr 28, 2025

rwgk left a comment

leofang left a comment

kkraus14 commented Apr 29, 2025

leofang commented Apr 29, 2025

github-actions bot commented Apr 29, 2025

CI: Run some tests with compute-sanitizer #566

CI: Run some tests with compute-sanitizer #566

Conversation

carterbox commented Apr 22, 2025 • edited Loading

Description

Checklist

copy-pr-bot bot commented Apr 22, 2025

leofang commented Apr 22, 2025

cryos commented Apr 22, 2025

carterbox commented Apr 22, 2025

leofang commented Apr 22, 2025

This comment has been minimized.

carterbox commented Apr 23, 2025

carterbox commented Apr 23, 2025

carterbox commented Apr 23, 2025 • edited Loading

carterbox commented Apr 24, 2025

carterbox commented Apr 24, 2025

carterbox commented Apr 25, 2025

carterbox commented Apr 25, 2025

copy-pr-bot bot commented Apr 25, 2025

leofang commented Apr 26, 2025

carterbox commented Apr 28, 2025

carterbox commented Apr 28, 2025

rwgk left a comment

Choose a reason for hiding this comment

carterbox commented Apr 28, 2025

rwgk left a comment

Choose a reason for hiding this comment

leofang left a comment

Choose a reason for hiding this comment

kkraus14 commented Apr 29, 2025

leofang commented Apr 29, 2025

github-actions bot commented Apr 29, 2025

carterbox commented Apr 22, 2025 •

edited

Loading

carterbox commented Apr 23, 2025 •

edited

Loading