Error in Evaluation after Applying PV-Tuning to AQLM Quantized Model

[gunho1123]

Hello,
I would like to evaluate a model that has been quantized using AQLM and subsequently fine-tuned with PV-Tuning.
After PV-Tuning, I obtained a best_model directory containing files such as model.layers.0.mlp.down_proj.weight.pth.

However, when I attempted to run lmeval.py, I encountered the following error:

FileNotFoundError: [Errno 2] No such file or directory: '/mnt/temp/AQLM/finetuned/meta-llama/Llama-3.1-8B/best_model/0.pth'

Could you help me resolve this issue? Specifically:

1. Should the fine-tuned model files be converted to a specific format before evaluation?
2. Is there any additional step required to ensure compatibility with lmeval.py?

Thank you in advance for your assistance.

[Vahe1994]

Hello!
Sorry for the late answer. After PV-tuning finetuning, because it is using FSDP you should convert it to standard format using https://github.com/Vahe1994/AQLM/tree/pv-tuning?tab=readme-ov-file#4-exporting-the-saved-model. Sorry for confusion. I will try to add this into Readme documentation.