Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

qwen2vl support #1042

Open
wants to merge 13 commits into
base: main
Choose a base branch
from
Open

qwen2vl support #1042

wants to merge 13 commits into from

Conversation

eaidova
Copy link
Collaborator

@eaidova eaidova commented Nov 29, 2024

What does this PR do?

Enable qwen2vl models export and inference with openvino.
Example of usage:

import requests
from PIL import Image
from transformers import AutoProcessor

from optimum.intel.openvino import OVModelForVisualCausalLM


model_id = "Qwen/Qwen2-VL-2B-Instruct"
# Load the model in half-precision on the available device(s)
model = OVModelForVisualCausalLM.from_pretrained(model_id)
processor = AutoProcessor.from_pretrained(model_id)

# Image
url = "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg"
image = Image.open(requests.get(url, stream=True).raw)

conversation = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
            },
            {"type": "text", "text": "Describe this image."},
        ],
    }
]


# Preprocess the inputs
text_prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)
# Excepted output: '<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\n<|vision_start|><|image_pad|><|vision_end|>Describe this image.<|im_end|>\n<|im_start|>assistant\n'

inputs = processor(text=[text_prompt], images=[image], padding=True, return_tensors="pt")

# Inference: Generation of the output
output_ids = model.generate(**inputs, max_new_tokens=10)
generated_ids = [output_ids[len(input_ids) :] for input_ids, output_ids in zip(inputs.input_ids, output_ids)]
output_text = processor.batch_decode(generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=True)
print(output_text)

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • Did you make sure to update the documentation with your changes?
  • Did you write any new necessary tests?

@eaidova eaidova force-pushed the ea/qwen2vl branch 2 times, most recently from ce7789f to 5af0206 Compare December 4, 2024 05:44
@HuggingFaceDocBuilderDev

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@AlexKoff88
Copy link
Collaborator

I don't have major comments. Thanks @eaidova!

@echarlaix, @IlyasMoutawwakil, the PR is ready for your review.

Copy link
Collaborator

@echarlaix echarlaix left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the addition @eaidova !

optimum/intel/openvino/modeling_visual_language.py Outdated Show resolved Hide resolved
optimum/intel/openvino/modeling_visual_language.py Outdated Show resolved Hide resolved
optimum/intel/openvino/modeling_visual_language.py Outdated Show resolved Hide resolved
optimum/exporters/openvino/model_patcher.py Show resolved Hide resolved
optimum/exporters/openvino/model_configs.py Outdated Show resolved Hide resolved
optimum/exporters/openvino/model_configs.py Outdated Show resolved Hide resolved
optimum/exporters/openvino/model_patcher.py Show resolved Hide resolved
@eaidova
Copy link
Collaborator Author

eaidova commented Dec 17, 2024

@echarlaix could you please check one more time?
Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants