MixEval-X Image / Video #434
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Luodian
approved these changes
Dec 3, 2024
| @@ -15,6 +15,7 @@ | |||
| from lmms_eval.api.instance import Instance | |||
| from lmms_eval.api.model import lmms | |||
| from lmms_eval.api.registry import register_model | |||
| from lmms_eval.models.model_utils.load_video import read_video_pyav_pil | |||
There was a problem hiding this comment.
This currently breaks as read_video_pyav_pil doesn't exist @pufanyi
There was a problem hiding this comment.
Hello, sorry about this. It seems that this change was accidentally brought over from another branch. Thank you for pointing it out! I've reverted it in #447.
ZhaoCinyu
pushed a commit
to ZhaoCinyu/lmms-eval
that referenced
this pull request
Dec 9, 2024
* reformat mix eval * video2text fix * video * Add image-to-text evaluation tasks and templates * Refactor image-to-text evaluation tasks and update configurations * Enhance LlamaVision model with video loading improvements and configuration updates * fix internvl2 * llava vid default numframe too small * add max_new_tokens * remove "with a short phrase" for `gpt4v`
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Image (LLaVA-1.5-13B)
official: 50.2
Video (LLaVA-Next-Video-7B)
official: 58.7