Support for Vision models

[Syst3m1cAn0maly]

I tried to quantize phi-3-vision with AutoFP8, but it didn't work and I got some errors in the process.:

Quantizing weights: 100%|██████████| 726/726 [00:00<00:00, 7888.09it/s]
Calibrating activation scales: 100%|██████████| 512/512 [02:33<00:00,  3.33it/s]
---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
Cell In[1], line 21
     19 # Load the model, quantize, and save checkpoint
     20 model = AutoFP8ForCausalLM.from_pretrained(pretrained_model_dir, quantize_config)
---> 21 model.quantize(examples)
     22 model.save_quantized(quantized_model_dir)

File [~/AutoFP8/auto_fp8/modeling.py:126](http://slnxiageneds02.marc.fr.ssg:8888/lab/tree/AutoFP8/auto_fp8/modeling.py#line=125), in AutoFP8ForCausalLM.quantize(self, calibration_tokens)
    123         return calibration_tokens.input_ids
    124     return calibration_tokens
--> 126 quantize_activations(
    127     self.model,
    128     self.quantize_config,
    129     _prepare_calibration_data(calibration_tokens),
    130 )

File [~/AutoFP8/auto_fp8/quantize.py:309](http://slnxiageneds02.marc.fr.ssg:8888/lab/tree/AutoFP8/auto_fp8/quantize.py#line=308), in quantize_activations(model, quantize_config, calibration_tokens)
    307 for k_proj_name, v_proj_name in kv_proj_pairs:
    308     parent_module_name = ".".join(k_proj_name.split(".")[:-1])
--> 309     assert parent_module_name == ".".join(v_proj_name.split(".")[:-1])
    310     parent_module = dict(model.named_modules())[parent_module_name]
    312     k_proj = dict(model.named_modules())[k_proj_name]

AssertionError:

Is there some specific setting for Vision models or is it not supported yet ?

Thanks in advance.