Skip to content

Support Qwen2_5_VLForConditionalGeneration #12595

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Conversation

csabakecskemeti
Copy link
Contributor

Make sure to read the contributing guidelines before submitting a PR

Based on this discussion:
#11483 (comment)

for the language part we can use the same architecture as Qwen2VLForConditionalGeneration.

Additionally HimariO has the update for the qwen2_vl_surgery.py to support --model_type "qwen2.5vl"
https://github.com/HimariO/llama.cpp.qwen2vl/tree/qwen25-vl
python examples/llava/qwen2_vl_surgery.py <path_to_model> --data_type fp32 --model_type "qwen2.5vl"

Tested working, examples:
DevQuasar/Qwen.Qwen2.5-VL-7B-Instruct-GGUF

build/bin/llama-qwen2vl-cli -m ~/Downloads/Qwen.Qwen2.5-VL-3B-Instruct.Q4_K_M.gguf --mmproj qwen.qwen2.5-vl-3b-instruct-vision.gguf -p "Describe this image." --image ~/Downloads/flower.jpg

encode_image_with_clip: image encoded in  3824.77 ms by CLIP (   54.64 ms per image patch)

The image depicts a flower with a unique and surreal appearance. The flower has a predominantly purple hue with a gradient effect, transitioning from deep purple to lighter shades as it moves towards the center. The petals are somewhat elongated and have a wavy, fluid texture, giving the flower a sense of movement or transformation. The center of the flower is a bright, electric blue, which contrasts sharply with the surrounding petals. The flower is set against a background that appears to be a blurred, soft green, which complements the purple and blue tones of the flower. Overall, the image has an artistic and somewhat otherworldly quality, with a focus on the vivid colors and intricate details of the flower.
llama_perf_context_print:        load time =   11328.57 ms

@github-actions github-actions bot added the python python script changes label Mar 26, 2025
@ngxson ngxson merged commit d5c6309 into ggml-org:master Mar 27, 2025
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
python python script changes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants