FATAL Failed to read file: , 314, , errno: , 2, , msg: , No such file or directory

[Hudayday]

Hi, I am running the readme_example.py using one A6000 GPUs. Everything follows the default setting.

However, when generating token, I get:
Model create: 99%|██████████████████████████████████████████▍| 4514/4578 [00:16<00:00, 2387.87it/s]2024-11-07 00:27:39.907 FATAL Failed to read file: , 314, , errno: , 2, , msg: , No such file or directory
terminate called after throwing an instance of 'std::runtime_error'
what(): Logged a FATAL error
Aborted (core dumped)

Looks like the files cannot be read.

I can see the offload files in the offload path.

[drunkcoding]

I am not able to reproduce the issue, have you tried to clean up the offloading dir and try again?