Getting no kernel image error

[ankur6ue]

Pytorch version: 2.5
torchrec version: 1
Cuda: 11.8
GPU: Nvidia 1080 Ti. Architecture: 6.1

Running the two tower example. Getting the following runtime error:

File "/home/ankur/miniconda3/envs/py3_12/lib/python3.12/site-packages/torchrec/sparse/jagged_tensor.py", line 79, in _to_offsets
return torch.ops.fbgemm.asynchronous_complete_cumsum(lengths)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ankur/miniconda3/envs/py3_12/lib/python3.12/site-packages/torch/_ops.py", line 1116, in call
return self._op(*args, **(kwargs or {}))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: CUDA error: no kernel image is available for execution on the device

Any suggestions?

[ankur6ue]

Also, I ran Pytorch's Distributed Dataparallel example on my computer just fine.. implying that the CUDA error is because of some cuda kernel binaries in the torchrec distribution?

[dstaay-fb]

Hi is this still an open issue?

[ankur6ue]

Yes it is! Any suggested fixes?