Error Running HOOMD-blue Script with CUDA Device

[idop11]

Hello,
I recently started using the HOOMD-blue package, which I installed via conda-forge. However, I encountered an issue while running two short and similar scripts, which I have attached as "script_without_error.txt" and "script_with_error.txt."

The first script ran without any problems, while the second script encountered an issue. The only difference between the two scripts is the absence of the line fire.forces.append(dpdc) in the first script.

Here is the error message I received when running the second script:

Traceback (most recent call last):
File "/mnt/c/Users/User/PycharmProjects/graphicsExperiments/script_with_error.py", line 55, in
simulation.run(100)
File "/home/idop/miniconda3/envs/hoomd_v4.4.1_new/lib/python3.12/site-packages/hoomd/simulation.py", line 555, in run
self.operations._schedule()
File "/home/idop/miniconda3/envs/hoomd_v4.4.1_new/lib/python3.12/site-packages/hoomd/operations.py", line 211, in _schedule
self.integrator._attach(sim)
File "/home/idop/miniconda3/envs/hoomd_v4.4.1_new/lib/python3.12/site-packages/hoomd/operation.py", line 315, in _attach
self._attach_hook()
File "/home/idop/miniconda3/envs/hoomd_v4.4.1_new/lib/python3.12/site-packages/hoomd/md/minimize/fire.py", line 261, in _attach_hook
super()._attach_hook()
File "/home/idop/miniconda3/envs/hoomd_v4.4.1_new/lib/python3.12/site-packages/hoomd/md/integrate.py", line 48, in _attach_hook
self._forces._sync(self._simulation, self._cpp_obj.forces)
File "/home/idop/miniconda3/envs/hoomd_v4.4.1_new/lib/python3.12/site-packages/hoomd/data/syncedlist.py", line 246, in _sync
raise err
File "/home/idop/miniconda3/envs/hoomd_v4.4.1_new/lib/python3.12/site-packages/hoomd/data/syncedlist.py", line 242, in _sync
self._register_item(item)
File "/home/idop/miniconda3/envs/hoomd_v4.4.1_new/lib/python3.12/site-packages/hoomd/data/syncedlist.py", line 190, in _register_item
value._attach(self._simulation)
File "/home/idop/miniconda3/envs/hoomd_v4.4.1_new/lib/python3.12/site-packages/hoomd/operation.py", line 315, in _attach
self._attach_hook()
File "/home/idop/miniconda3/envs/hoomd_v4.4.1_new/lib/python3.12/site-packages/hoomd/md/pair/pair.py", line 143, in _attach_hook
self._cpp_obj = cls(self._simulation.state._cpp_sys_def,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: CUDA Error: invalid device ordinal before /hoomd/Autotuner.h:498

Does anyone have any insights into the reason for this problem?

By the way, I explicitly included hoomd.device.GPU(gpu_ids=[0]) because I only have one GPU, and I read that it might solve the problem. Unfortunately, it didn't.

Thanks,
Ido
script_without_error.txt
script_with_error.txt

[joaander]

Your script_with_error.txt runs without error on my system. What is the output of nvidia-smi on yours?

[idop11]

This is the output I received (I'm not very familiar with these issues, so I apologize if I should have understood the problem immediately):

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.40.06              Driver Version: 551.23         CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce MX250           On  |   00000000:01:00.0 Off |                  N/A |
| N/A   60C    P8             N/A / ERR!  |       0MiB /   2048MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A        32      G   /Xwayland                                   N/A      |
+-----------------------------------------------------------------------------------------+```

[joaander]

The MX250 meets the minimum requirements to run HOOMD: https://www.techpowerup.com/gpu-specs/geforce-mx250.c3353 (compute 6.0 or newer). Please be aware that I have never tested HOOMD on laptop class GPU hardware.

Invalid device ordinal may not be the first error. Please modify your script to enable GPU error checking and report the output.

gpu = hoomd.device.GPU(gpu_ids=[0])
gpu.gpu_error_checking = True

[idop11]

I modified the script in the following way, and it returned the same output, if I understand it correctly:

Traceback (most recent call last):
  File "/mnt/c/Users/User/PycharmProjects/graphicsExperiments/script_with_error.py", line 56, in <module>
    simulation.run(100)
  File "/home/idop/miniconda3/envs/hoomd_v4.4.1_new/lib/python3.12/site-packages/hoomd/simulation.py", line 555, in run
    self.operations._schedule()
  File "/home/idop/miniconda3/envs/hoomd_v4.4.1_new/lib/python3.12/site-packages/hoomd/operations.py", line 211, in _schedule
    self.integrator._attach(sim)
  File "/home/idop/miniconda3/envs/hoomd_v4.4.1_new/lib/python3.12/site-packages/hoomd/operation.py", line 315, in _attach
    self._attach_hook()
  File "/home/idop/miniconda3/envs/hoomd_v4.4.1_new/lib/python3.12/site-packages/hoomd/md/minimize/fire.py", line 261, in _attach_hook
    super()._attach_hook()
  File "/home/idop/miniconda3/envs/hoomd_v4.4.1_new/lib/python3.12/site-packages/hoomd/md/integrate.py", line 48, in _attach_hook
    self._forces._sync(self._simulation, self._cpp_obj.forces)
  File "/home/idop/miniconda3/envs/hoomd_v4.4.1_new/lib/python3.12/site-packages/hoomd/data/syncedlist.py", line 246, in _sync
    raise err
  File "/home/idop/miniconda3/envs/hoomd_v4.4.1_new/lib/python3.12/site-packages/hoomd/data/syncedlist.py", line 242, in _sync
    self._register_item(item)
  File "/home/idop/miniconda3/envs/hoomd_v4.4.1_new/lib/python3.12/site-packages/hoomd/data/syncedlist.py", line 190, in _register_item
    value._attach(self._simulation)
  File "/home/idop/miniconda3/envs/hoomd_v4.4.1_new/lib/python3.12/site-packages/hoomd/operation.py", line 315, in _attach
    self._attach_hook()
  File "/home/idop/miniconda3/envs/hoomd_v4.4.1_new/lib/python3.12/site-packages/hoomd/md/pair/pair.py", line 143, in _attach_hook
    self._cpp_obj = cls(self._simulation.state._cpp_sys_def,
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: CUDA Error: invalid device ordinal before /hoomd/Autotuner.h:498

The traceback is at line 56 now because I added the line gpu.gpu_error_checking = True

[joaander]

Indeed. The CUDA error is the same. I don't know what would cause this.

[fergusonml]

I got something similar using windows 10 wsl2 and condaforge.

$ python3 -m hoomd_benchmarks.md_pair_lj --device GPU
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/ferguson/hoomd-benchmarks/hoomd_benchmarks/md_pair_lj.py", line 24, in <module>
    MDPairLJ.main()
  File "/home/ferguson/hoomd-benchmarks/hoomd_benchmarks/common.py", line 219, in main
    performance = benchmark.execute()
                  ^^^^^^^^^^^^^^^^^^^
  File "/home/ferguson/hoomd-benchmarks/hoomd_benchmarks/common.py", line 116, in execute
    self.run(0)
  File "/home/ferguson/hoomd-benchmarks/hoomd_benchmarks/common.py", line 105, in run
    self.sim.run(steps)
  File "/home/ferguson/anaconda3/envs/hoomd/lib/python3.12/site-packages/hoomd/simulation.py", line 555, in run
    self.operations._schedule()
  File "/home/ferguson/anaconda3/envs/hoomd/lib/python3.12/site-packages/hoomd/operations.py", line 211, in _schedule
    self.integrator._attach(sim)
  File "/home/ferguson/anaconda3/envs/hoomd/lib/python3.12/site-packages/hoomd/operation.py", line 315, in _attach
    self._attach_hook()
  File "/home/ferguson/anaconda3/envs/hoomd/lib/python3.12/site-packages/hoomd/md/integrate.py", line 293, in _attach_hook
    super()._attach_hook()
  File "/home/ferguson/anaconda3/envs/hoomd/lib/python3.12/site-packages/hoomd/md/integrate.py", line 46, in _attach_hook
    self._forces._sync(self._simulation, self._cpp_obj.forces)
  File "/home/ferguson/anaconda3/envs/hoomd/lib/python3.12/site-packages/hoomd/data/syncedlist.py", line 246, in _sync
    raise err
  File "/home/ferguson/anaconda3/envs/hoomd/lib/python3.12/site-packages/hoomd/data/syncedlist.py", line 242, in _sync
    self._register_item(item)
  File "/home/ferguson/anaconda3/envs/hoomd/lib/python3.12/site-packages/hoomd/data/syncedlist.py", line 190, in _register_item
    value._attach(self._simulation)
  File "/home/ferguson/anaconda3/envs/hoomd/lib/python3.12/site-packages/hoomd/operation.py", line 315, in _attach
    self._attach_hook()
  File "/home/ferguson/anaconda3/envs/hoomd/lib/python3.12/site-packages/hoomd/md/pair/pair.py", line 143, in _attach_hook
    self._cpp_obj = cls(self._simulation.state._cpp_sys_def,
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: CUDA Error: invalid device ordinal before /hoomd/Autotuner.h:498

$ nvidia-smi
Mon Mar 18 11:37:53 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.60.01              Driver Version: 551.76         CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 3080 Ti     On  |   00000000:01:00.0  On |                  N/A |
| 35%   41C    P8             29W /  350W |     784MiB /  12288MiB |      2%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A      5325      C   /python3.12                                 N/A      |
+-----------------------------------------------------------------------------------------+

[fergusonml]

$ python3 -m hoomd_benchmarks --device GPU
HPMCSphere: 445.7145327674839
Segmentation fault
$ python3 -m hoomd_benchmarks --device CPU
HPMCSphere: 42.4472810075796

[joaander]

Windows does not support unified memory. You must run HOOMD-blue on a native Linux OS for GPU support.