Llava models support

[iKachaykin]

Hey! Thanks a lot for your amazing package! I would like to run Llava models with candle, but I am a bit confused right now how may I do that. Please correct me if I am wrong: to run any custom model not listed in candle-examples, I need to reimplement it from scratch using candle, correct? Or maybe there is any other better way to run a .gguf file with candle?

[ZizoAdam]

Trying to work through this myself now, haven't touched the framework before so I'm probably as lost as you are. Only thing I've found I can do with a GGUF so far thanks to reading gguf_file.rs is reading it like this

    let model_path = PathBuf::from("models//mistral-7b-instruct-v0.2.Q5_K_M.gguf");
    let model_file = std::fs::File::open(model_path)?;
    let mmap = unsafe { memmap2::Mmap::map(&model_file)? };
    let cursor = std::io::Cursor::new(mmap);
    let mut reader = std::io::BufReader::new(cursor);
    let gguf = Content::read(&mut reader)?;

[EricLBuehler]

It looks like the Llava model is based on a Mistral architecture, so it should be able to run with any Mistral architecture? Perhaps the quantized example or mistral.rs could be used to run the weights.