[Feature] Support llava onevision #2783
Conversation
deepindeed2022
changed the title
docs/zh_cn/multi_modal/llava.md
Outdated
| | llava-hf/llava-1.5-7b-hf | 7B | TurboMind, PyTorch | | ||
| | liuhaotian/llava-v1.6-vicuna-7b | 7B | TurboMind, PyTorch | | ||
| | liuhaotian/llava-v1.6-mistral-7b | 7B | TurboMind, PyTorch | | ||
| | lmms-lab/llava-onevision-qwen2-7b-ov | 0.5B,7B,72B | TurboMind, PyTorch | |
There was a problem hiding this comment.
Hi, the model is not supported by pytorch engine as it fails with pipeline.
lmdeploy/lmdeploy/pytorch/models/patch.py", line 177, in _get_model_class
raise RuntimeError(
RuntimeError: Can not found rewrite for architectures: ['LlavaQwenForCausalLM']
There was a problem hiding this comment.
delete PyTorchBackend
deepindeed2022
force-pushed
the
support_llava_onevision
branch
from
c15538a
to
948fffe
Compare
| @@ -23,7 +28,7 @@ def check_llava_install(): | |||
| except ImportError: | |||
| raise ImportError( | |||
| 'To use LlavaVLModel, please install llava by ' | |||
| 'pip install git+https://github.com/haotian-liu/LLaVA.git --no-deps' # noqa: E501 | |||
| 'pip install git+https://github.com/LLaVA-VL/LLaVA-NeXT.git --no-deps' # noqa: E501 | |||
There was a problem hiding this comment.
Will this PR affect the deployment of liuhaotian/llava-v1.5-7b
There was a problem hiding this comment.
The model worked well on the new repo
|
Hi, @deepindeed2022 thanks for your contribution to LMDeploy |
Yes, I 'll follow up after this PR #2810 merge. |
| f'{self.hf_config.vision_feature_select_strategy}') | ||
| image_features = self.model.multi_modal_projector(image_features) | ||
| image_features = torch.split(image_features, image_num_patches, dim=0) | ||
| image_features = self.model.get_image_features( |
There was a problem hiding this comment.
hi, it seems llava-next models do not have this method for transformers==4.45.2
File "/lmdeploy/lmdeploy/vl/model/llava_next.py", line 68, in forward
image_features = self.model.get_image_features(
File "/opt/py3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1729, in __getattr__
raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")
AttributeError: 'LlavaNextForConditionalGeneration' object has no attribute 'get_image_features'
There was a problem hiding this comment.
the new feature in recently release transformers(about 1 month), I use transformers==4.46.3.
There was a problem hiding this comment.
maybe we could check transformers version in the initialization of llavanext and give info of using transformers>=4.60.0 if not met
Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily receiving feedbacks. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.
Motivation
Support llava-onevision with LMDeploy
Modification
Use cases (Optional)
Checklist