"facebook/opt-125m" and other sub 1B parameter LM's?

[dcompgriff]

I see that the code has options for using small OPT LM's. Will you release weights for VLM's trained with these smaller LM models?

[zhangshaolei1998]

Thanks for your question.
The "small OPT LM" appears due to some settings in the original LLaVA code. We did not train the related LLaVA-Mini on smaller LMs.
If necessary, you can train an LLaVA-Mini based on smaller LMs, which will be more efficient.

[dcompgriff]

Where are the training and fine-tuning scripts you use. I see scripts/llavamini/train.sh, but no script for fine-tuning that includes the "During instruction tuning, we combine 665K image instruction data from LLaVA (Liu et al., 2023b), 100K video instruction data from Video-ChatGPT" that the paper mentions. Do you just start with 'scripts/pretrain.sh' as your stage 1 with your "ICTNLP/llava-mini-llama-3-8b" as the model path? It's not clear because the --vision_tower parameters don't match for those two files.