RuntimeError: CUDA error: invalid configuration argument Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

[1999XIAOZHANG]

Why does it work without errors when I use Replica, but when I use my own dataset, this issue occurs? What is causing it? My dataset contains only 700 images with a resolution of 760x1280.

(hislam2) root@baf545cb2c29:/HI-SLAM2# python demo.py --imagedir /HI-SLAM2/data/mydata/images --calib /HI-SLAM2/data/mydata/calib.txt --config /HI-SLAM2/config/owndata_config.yaml --output outputs/owndata --undistort
Processing keyframe 11 gs 0: 6%|██████▋ | 48/763 [00:30<01:04, 11.05it/s]GSBackend: Initialized map
Processing keyframe 55 gs 100791: 46%|██████████████████████████████████████████████▊ | 354/763 [03:07<05:31, 1.23it/s]GSBackend: Resetting the opacity of non-visible Gaussians
Processing keyframe 103 gs 99477: 87%|███████████████████████████████████████████████████████████████████████████████████████▍ | 661/763 [05:36<00:44, 2.31it/s]Traceback (most recent call last):
File "/HI-SLAM2/demo.py", line 120, in
hi2.track(t, image, intrinsics=intrinsics, is_last=is_last)
File "/HI-SLAM2/hislam2/hi2.py", line 103, in track
self.call_gs(viz_idx)
File "/HI-SLAM2/hislam2/hi2.py", line 83, in call_gs
self.gs.process_track_data(data)
File "/HI-SLAM2/hislam2/gs_backend.py", line 113, in process_track_data
self.add_next_kf(idx, viewpoint, depth_map=packet["depths"][i].numpy())
File "/HI-SLAM2/hislam2/gs_backend.py", line 153, in add_next_kf
self.gaussians.extend_from_pcd_seq(
File "/HI-SLAM2/hislam2/gaussian/scene/gaussian_model.py", line 221, in extend_from_pcd_seq
self.extend_from_pcd(
File "/HI-SLAM2/hislam2/gaussian/scene/gaussian_model.py", line 204, in extend_from_pcd
self.densification_postfix(
File "/HI-SLAM2/hislam2/gaussian/scene/gaussian_model.py", line 456, in densification_postfix
optimizable_tensors = self.cat_tensors_to_optimizer(d)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/HI-SLAM2/hislam2/gaussian/scene/gaussian_model.py", line 409, in cat_tensors_to_optimizer
stored_state["exp_avg"] = torch.cat(
^^^^^^^^^^
RuntimeError: CUDA error: invalid configuration argument
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

[Willyzw]

Sorry to hear about the issue! It seems related to #3 , and the error might originate from the densification process. This could potentially be caused by NaNs or zero-length tensors that cannot be concatenated, possibly due to incorrect tracking data. Did you maybe use the preprocess_scannet.py to prepare data? For better understanding could you rerun the code with the CUDA_LAUNCH_BLOCKING=1 prefix to confirm if the error actually occurs at the reported line. Additionally, it would to helpful to inspect the tensors using the following code snippet to check for anomalies in densification_postfix function

for k, v in d.items():
    print(k, v.shape, torch.isnan(v).any(), torch.isinf(v).any())

This should provide more insights wht is going wrong