Feature Request - Support for vision models like QwenVL and CogVLM

[RYG81]

Hey, it would be great if you guys can add nodes for using QwenVL (https://huggingface.co/Qwen/Qwen-VL) and CogVLM (https://github.com/THUDM/CogVLM) models, this will be very useful for people who want to use good vision models but don't want to go for OpenAI or Gemini or other paid services.

[shhlife]

@collindutter how do you think we can use these? the Qwen-VL - would that work with the HuggingFaceHubPromptDriver? or is this a new type of task we'd need to create?

I think with the CogVLM model, we would be able to use Ollama to do it, if that were available.. @RYG81 there are other vision models available for Ollama - like llava or or llama-3.2-vision.. could you use them?

[collindutter]

HuggingFaceHubPromptDriver does not support Image inputs yet. I will create a ticket to track this.

Anything on Ollama would probably be the best solution today.