FileNotFoundError: [Errno 2] No such file or directory: 'checkpoint-100/model-00001-of-00003.safetensors'

[dshwei]

when mutil gpu run starcoder in full parameter tuning ,

File "starcoder-git/finetune.py", line 44, in on_save
kwargs["model"].save_pretrained(checkpoint_folder)
File "/miniconda3/envs/sqlcode/lib/python3.9/site-packages/transformers/modeling_utils.py", line 2480, in save_pretrained
os.remove(full_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'checkpoint-100/model-00001-of-00003.safetensors'

this error maybe when mutil pIds process checkpoints in saved directory