CUDA error: an illegal memory access was encountered

[zhaojiancheng007]

Thanks for your great work; it's very interesting. When I was testing the code, I often encountered issues with illegal CUDA memory access during program execution. Have you encountered this problem? What could be causing this?"
Thanks !

RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

[Xinjie-Q]

Could you provide the running command? When I run the commands from the script, I do not have such a problem.

[zhaojiancheng007]

Hi ! here is my running script

for num_points in 1000 1500 2000
do
CUDA_VISIBLE_DEVICES=0 python train.py -d $data_path
--data_name beauty --model_name GaussianImage_RS --num_points $num_points --iterations 30000 --save_imgs
done

This error occored kind of random, but more frequent when I tested gaussians Image using relatively high resolution data, for example 1920 x 960, or 1280 x 960.

[ChaosAdmStudent]

This issue happens when you compile gsplat during first run (JIT). When I pip install gsplat by compiling CUDA libraries on installation, everything works for me. Maybe something to try out.