diff --git a/README.md b/README.md index 7be1c6c7..45e0896c 100755 --- a/README.md +++ b/README.md @@ -4,13 +4,22 @@ # The Evaluation Suite of Large Multimodal Models +[![PyPI](https://img.shields.io/pypi/v/lmms-eval)](https://pypi.org/project/lmms-eval) +![PyPI - Downloads](https://img.shields.io/pypi/dm/lmms-eval) +![GitHub contributors](https://img.shields.io/github/contributors/EvolvingLMMs-Lab/lmms-eval) +[![issue resolution](https://img.shields.io/github/issues-closed-raw/EvolvingLMMs-Lab/lmms-eval)](https://github.com/EvolvingLMMs-Lab/lmms-eval/issues) +[![open issues](https://img.shields.io/github/issues-raw/EvolvingLMMs-Lab/lmms-eval)](https://github.com/EvolvingLMMs-Lab/lmms-eval/issues) + > Accelerating the development of large multimodal models (LMMs) with `lmms-eval` -🏠 [LMMs-Lab Homepage](https://lmms-lab.github.io/) | πŸŽ‰ [Blog](https://lmms-lab.github.io/lmms-eval-blog/lmms-eval-0.1/) | πŸ“š [Documentation](docs/README.md) | πŸ€— [Huggingface Datasets](https://huggingface.co/lmms-lab) | Discord_Thread [discord/lmms-eval](https://discord.gg/zdkwKUqrPy) +🏠 [LMMs-Lab Homepage](https://lmms-lab.framer.ai) | πŸŽ‰ [Blog](https://lmms-lab.framer.ai/blog/lmms-eval-blog) | πŸ“š [Documentation](docs/README.md) | πŸ€— [Huggingface Datasets](https://huggingface.co/lmms-lab) | Discord_Thread [discord/lmms-eval](https://discord.gg/zdkwKUqrPy) --- ## Annoucement +- [2024-08] πŸŽ‰πŸŽ‰ We welcome the new model [LLaVA-OneVision](https://huggingface.co/papers/2408.03326), [Mantis](https://github.com/EvolvingLMMs-Lab/lmms-eval/pull/162), new tasks [MVBench](https://huggingface.co/datasets/OpenGVLab/MVBench), [LongVideoBench](https://github.com/EvolvingLMMs-Lab/lmms-eval/pull/117), [MMStar](https://github.com/EvolvingLMMs-Lab/lmms-eval/pull/158). We provide new feature of SGlang Runtime API for llava-onevision model, please refer the [doc](https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/main/docs/commands.md) for inference acceleration + +- [2024-07] πŸŽ‰πŸŽ‰ We have released the [technical report](https://arxiv.org/abs/2407.12772) and [LiveBench](https://huggingface.co/spaces/lmms-lab/LiveBench)! - [2024-07] πŸ‘¨β€πŸ’»πŸ‘¨β€πŸ’» The `lmms-eval/v0.2.1` has been upgraded to support more models, including [LongVA](https://github.com/EvolvingLMMs-Lab/LongVA), [InterVL-2](https://github.com/OpenGVLab/InternVL), [VILA](https://github.com/NVlabs/VILA), and many more evaluation tasks, e.g. [Details Captions](https://github.com/EvolvingLMMs-Lab/lmms-eval/pull/136), [MLVU](https://arxiv.org/abs/2406.04264), [WildVision-Bench](https://huggingface.co/datasets/WildVision/wildvision-arena-data), [VITATECS](https://github.com/lscpku/VITATECS) and [LLaVA-Interleave-Bench](https://llava-vl.github.io/blog/2024-06-16-llava-next-interleave/). @@ -48,7 +57,7 @@ cd lmms-eval pip install -e . ``` -If you wanted to test llava, you will have to clone their repo from [LLaVA](https://github.com/haotian-liu/LLaVA) and +If you want to test LLaVA, you will have to clone their repo from [LLaVA](https://github.com/haotian-liu/LLaVA) and ```bash # for llava 1.5 # git clone https://github.com/haotian-liu/LLaVA @@ -68,7 +77,7 @@ You can check the [environment install script](miscs/repr_scripts.sh) and [torch -If you want to test on caption dataset such as `coco`, `refcoco`, and `nocaps`, you will need to have `java==1.8.0 ` to let pycocoeval api to work. If you don't have it, you can install by using conda +If you want to test on caption dataset such as `coco`, `refcoco`, and `nocaps`, you will need to have `java==1.8.0` to let pycocoeval api to work. If you don't have it, you can install by using conda ``` conda install openjdk=8 ``` @@ -92,6 +101,11 @@ We also provide the raw data exported from Weights & Biases for the detailed res
+If you want to test [VILA](https://github.com/NVlabs/VILA), you should install the following dependencies: + +```bash +pip install s2wrapper@git+https://github.com/bfshi/scaling_on_scales +``` Our Development will be continuing on the main branch, and we encourage you to give us feedback on what features are desired and how to improve the library further, or ask questions, either in issues or PRs on GitHub. @@ -165,6 +179,18 @@ python3 -m accelerate.commands.launch \ python3 -m accelerate.commands.launch --num_processes=8 -m lmms_eval --config ./miscs/example_eval.yaml ``` +**Evaluation of video model (llava-next-video-32B)** +```bash +accelerate launch --num_processes 8 --main_process_port 12345 -m lmms_eval \ + --model llavavid \ + --model_args pretrained=lmms-lab/LLaVA-NeXT-Video-32B-Qwen,conv_template=qwen_1_5,video_decode_backend=decord,max_frames_num=32，mm_spatial_pool_mode=average,mm_newline_position=grid,mm_resampler_location=after \ + --tasks videomme \ + --batch_size 1 \ + --log_samples \ + --log_samples_suffix llava_vid_32B \ + --output_path ./logs/ +``` + **Evaluation with naive model sharding for bigger model (llava-next-72b)** ```bash @@ -199,7 +225,7 @@ Please check [supported models](lmms_eval/models/__init__.py) for more details. ### Supported tasks -Please check [supported tasks](lmms_eval/docs/current_tasks.md) for more details. +Please check [supported tasks](docs/current_tasks.md) for more details. ## Add Customized Model and Dataset diff --git a/docs/commands.md b/docs/commands.md index e48ca36f..0455d05a 100755 --- a/docs/commands.md +++ b/docs/commands.md @@ -193,3 +193,50 @@ pip install flashinfer -i https://flashinfer.ai/whl/cu121/torch2.3/ ``` +## Usage with SRT API + +> install sglang + +```bash +git clone https://github.com/sgl-project/sglang.git +# Current version is tested on #1222 +cd sglang; +pip install -e "python[srt]" + +# Install FlashInfer CUDA kernels +pip install flashinfer -i https://flashinfer.ai/whl/cu121/torch2.4/ +``` + +> run sglang backend service with the following command + +```bash +# After update, there is no need to use an extra command to setup backend server +# the server will be initialized in the init process + +# launch lmms-eval srt_api model +CKPT_PATH=$1 +TASK=$2 +MODALITY=$3 +TP_SIZE=$4 +echo $TASK +TASK_SUFFIX="${TASK//,/_}" +echo $TASK_SUFFIX + +python3 -m lmms_eval \ + --model srt_api \ + --model_args modality=$MODALITY,model_version=$CKPT_PATH,tp=$TP_SIZE,host=127.0.0.1,port=30000,timeout=600 \ + --tasks $TASK \ + --batch_size 1 \ + --log_samples \ + --log_samples_suffix $TASK_SUFFIX \ + --output_path ./logs/ +``` + +You may need to install some dependencies for the above command to work (if you encounter some errors). + +```bash +pip install httpx==0.23.3 +pip install protobuf==3.20 +``` + + diff --git a/docs/current_tasks.md b/docs/current_tasks.md index 6e0e5238..a9949cb6 100644 --- a/docs/current_tasks.md +++ b/docs/current_tasks.md @@ -105,6 +105,7 @@ - ScreenSpot REG / Instruction Generation (screenspot_reg) - SeedBench (seedbench) - SeedBench 2 (seedbench_2) +- SeedBench 2 Plus (seedbench_2_plus) - ST-VQA (stvqa) - TextCaps (textcaps) - TextCaps Validation (textcaps_val) diff --git a/docs/task_guide.md b/docs/task_guide.md index 1376bc22..1e7d3a9d 100755 --- a/docs/task_guide.md +++ b/docs/task_guide.md @@ -40,7 +40,7 @@ metric_list: - metric: mme_cognition_score aggregation: !function utils.mme_aggregate_results higher_is_better: true -model_specific_prompt_kwargs: +lmms_eval_specific_kwargs: default: pre_prompt: "" post_prompt: "\nAnswer the question using a single word or phrase." @@ -52,7 +52,7 @@ metadata: ``` You can pay special attention to the `process_results` and `metric_list` fields, which are used to define how the model output is post-processed and scored. -Also, the `model_specific_prompt_kwargs` field is used to define model-specific prompt configurations. The default is set to follow Llava. +Also, the `lmms_eval_specific_kwargs` field is used to define model-specific prompt configurations. The default is set to follow Llava. PPL-based tasks: - Seedbench (`lmms_eval/tasks/seedbench/seedbench_ppl.yaml`) diff --git a/lmms_eval/api/task.py b/lmms_eval/api/task.py index 1f44e40e..cd9e90b6 100755 --- a/lmms_eval/api/task.py +++ b/lmms_eval/api/task.py @@ -861,6 +861,7 @@ def _download_from_youtube(path): if "video" in dataset_kwargs and dataset_kwargs["video"]: hf_home = os.getenv("HF_HOME", "~/.cache/huggingface/") + hf_home = os.path.expanduser(hf_home) cache_dir = dataset_kwargs["cache_dir"] cache_dir = os.path.join(hf_home, cache_dir) accelerator = Accelerator() @@ -955,13 +956,13 @@ def concat_tar_parts(tar_parts, output_tar): download_config=download_config, **dataset_kwargs if dataset_kwargs is not None else {}, ) - self.dataset_no_image = datasets.load_dataset( - path=self.DATASET_PATH, - name=self.DATASET_NAME, - download_mode=datasets.DownloadMode.REUSE_DATASET_IF_EXISTS, - download_config=download_config, - **dataset_kwargs if dataset_kwargs is not None else {}, - ) + if self.config.process_docs is not None: + for split in self.dataset: + if split in [self.config.training_split, self.config.validation_split, self.config.test_split, self.config.fewshot_split]: + self.dataset[split] = self.config.process_docs(self.dataset[split]) + + # copy dataset, remove image features + self.dataset_no_image = self.dataset.copy() for doc_name in self.dataset_no_image: remove_cols = [] features = self.dataset_no_image[doc_name].features @@ -994,20 +995,14 @@ def has_test_docs(self) -> bool: def training_docs(self) -> datasets.Dataset: if self.has_training_docs(): - if self.config.process_docs is not None: - return self.config.process_docs(self.dataset[self.config.training_split]) return self.dataset[self.config.training_split] def validation_docs(self) -> datasets.Dataset: if self.has_validation_docs(): - if self.config.process_docs is not None: - return self.config.process_docs(self.dataset[self.config.validation_split]) return self.dataset[self.config.validation_split] def test_docs(self) -> datasets.Dataset: if self.has_test_docs(): - if self.config.process_docs is not None: - return self.config.process_docs(self.dataset[self.config.test_split]) return self.dataset[self.config.test_split] def fewshot_docs(self): diff --git a/lmms_eval/evaluator.py b/lmms_eval/evaluator.py index 97cc994e..066fc0cd 100755 --- a/lmms_eval/evaluator.py +++ b/lmms_eval/evaluator.py @@ -604,8 +604,13 @@ def evaluate( } if log_samples: results_dict["samples"] = dict(samples) + else: + results_dict = None - return results_dict + with open(f"{cli_args.output_path}/rank{int(os.environ.get('RANK', 0))}_metric_eval_done.txt", "w") as f: + f.write(f"rank {int(os.environ.get('RANK', 0))} eval done") + while len([file for file in os.listdir(cli_args.output_path) if file.endswith("metric_eval_done.txt")]) < lm._world_size: + time.sleep(1) else: return None diff --git a/lmms_eval/models/internvl.py b/lmms_eval/models/internvl.py index 14ba59c2..c238bf59 100644 --- a/lmms_eval/models/internvl.py +++ b/lmms_eval/models/internvl.py @@ -455,7 +455,10 @@ def _collate(x): split = split[0] batched_visuals = [doc_to_visual[0](self.task_dict[task][split][ids]) for ids in doc_id] # [B, N] flattened_visuals = self.flatten(batched_visuals) - pixel_values = self.load_image(flattened_visuals, self.image_size).cuda().to(torch.bfloat16) + try: + pixel_values = self.load_image(flattened_visuals, self.image_size).cuda().to(torch.bfloat16) + except IndexError: + pixel_values = None gen_kwargs = all_gen_kwargs[0] if "max_new_tokens" not in gen_kwargs: diff --git a/lmms_eval/models/internvl2.py b/lmms_eval/models/internvl2.py index 3d66c0bf..fadf778e 100644 --- a/lmms_eval/models/internvl2.py +++ b/lmms_eval/models/internvl2.py @@ -148,6 +148,7 @@ def __init__( accelerator_kwargs = InitProcessGroupKwargs(timeout=timedelta(weeks=52)) accelerator = Accelerator(kwargs_handlers=[accelerator_kwargs]) + self.accelerator = accelerator if accelerator.num_processes > 1: self._device = torch.device(f"cuda:{accelerator.local_process_index}") self.device_map = f"cuda:{accelerator.local_process_index}" @@ -254,13 +255,16 @@ def generate_until(self, requests) -> List[str]: visuals = [doc_to_visual(self.task_dict[task][split][doc_id])] visuals = self.flatten(visuals) if self.modality == "image": - visuals = [load_image(visual).to(torch.bfloat16).cuda() for visual in visuals] - pixel_values = torch.cat(visuals, dim=0) - num_patches_list = [visual.size(0) for visual in visuals] if visuals: + visuals = [load_image(visual).to(torch.bfloat16).cuda() for visual in visuals] + pixel_values = torch.cat(visuals, dim=0) + num_patches_list = [visual.size(0) for visual in visuals] image_tokens = [""] * len(visuals) image_tokens = " ".join(image_tokens) contexts = image_tokens + "\n" + contexts + else: + pixel_values = None + num_patch_list = None response, history = self.model.chat(self.tokenizer, pixel_values, contexts, gen_kwargs, num_patches_list=num_patches_list, history=None, return_history=True) elif self.modality == "video": assert len(visuals) == 1, f"Only one video is supported, but got {len(visuals)} videos." diff --git a/lmms_eval/models/llava.py b/lmms_eval/models/llava.py index 8a7d7de8..9940bc76 100755 --- a/lmms_eval/models/llava.py +++ b/lmms_eval/models/llava.py @@ -73,6 +73,7 @@ def __init__( accelerator_kwargs = InitProcessGroupKwargs(timeout=timedelta(weeks=52)) accelerator = Accelerator(kwargs_handlers=[accelerator_kwargs]) + self.accelerator = accelerator if accelerator.num_processes > 1: self._device = torch.device(f"cuda:{accelerator.local_process_index}") self.device_map = f"cuda:{accelerator.local_process_index}" diff --git a/lmms_eval/models/llava_hf.py b/lmms_eval/models/llava_hf.py index fb0fadc0..0fab0e01 100644 --- a/lmms_eval/models/llava_hf.py +++ b/lmms_eval/models/llava_hf.py @@ -56,6 +56,7 @@ def __init__( device_map: str = "", chat_template: Optional[str] = None, use_cache: bool = True, + specified_eot_token_id: Optional[int] = None, **kwargs, ) -> None: super().__init__() @@ -89,6 +90,7 @@ def __init__( self.batch_size_per_gpu = int(batch_size) self.chat_template = chat_template self.use_cache = use_cache + self.specified_eot_token_id = specified_eot_token_id if accelerator.num_processes > 1 and device_map == "": assert accelerator.distributed_type in [DistributedType.FSDP, DistributedType.MULTI_GPU, DistributedType.DEEPSPEED], "Unsupported distributed type provided. Only DDP and FSDP are supported." # If you want to use DistributedType.DEEPSPEED, you have to run accelerate config before using the model @@ -320,18 +322,13 @@ def _collate(x): max_new_tokens=gen_kwargs["max_new_tokens"], use_cache=self.use_cache, pad_token_id=self.tokenizer.eos_token_id, + eos_token_id=self.specified_eot_token_id, ) + cont = cont[:, inputs["input_ids"].shape[-1] :] except Exception as e: eval_logger.error(f"Error {e} in generating") cont = "" text_outputs = self.tokenizer.batch_decode(cont, skip_special_tokens=True)[0] - if "1.5" in self.pretrained: - text_outputs = text_outputs.split("ASSISTANT:")[-1].strip() - elif "mistral" in self.pretrained: - text_outputs = text_outputs.split("[/INST]")[-1].strip() - else: - text_outputs = text_outputs.split("ASSISTANT:")[-1].strip() - if self.accelerator.is_main_process and doc_id[0] % 100 == 0: eval_logger.debug(f"Generated text for doc ID {doc_id[0]}:\n\n{text_outputs}\n") diff --git a/lmms_eval/models/llava_vid.py b/lmms_eval/models/llava_vid.py index 724a3cdb..2965be1e 100755 --- a/lmms_eval/models/llava_vid.py +++ b/lmms_eval/models/llava_vid.py @@ -109,6 +109,7 @@ def __init__( self.max_frames_num = int(max_frames_num) self.mm_resampler_location = mm_pooling_position self.delay_load = delay_load + if self.overwrite == True: overwrite_config = {} overwrite_config["mm_resampler_type"] = self.mm_resampler_type @@ -116,7 +117,7 @@ def __init__( overwrite_config["mm_spatial_pool_out_channels"] = self.mm_spatial_pool_out_channels overwrite_config["mm_spatial_pool_mode"] = self.mm_spatial_pool_mode overwrite_config["mm_pooling_position"] = self.mm_resampler_location - overwrite_config["mm_newline_position"] = mm_newline_position + overwrite_config["mm_newline_position"] = self.mm_newline_position overwrite_config["add_faster_video"] = False overwrite_config["delay_load"] = self.delay_load # overwrite_config["attn_implementation"] = attn_implementation @@ -145,7 +146,7 @@ def __init__( self._tokenizer = AutoTokenizer.from_pretrained(pretrained, use_fast=False) cfg_pretrained = LlavaConfig.from_pretrained(pretrained) if overwrite_config is not None: - print(f"Overwriting config with {overwrite_config}") + eval_logger.log(f"Overwriting config with {overwrite_config}") for k, v in overwrite_config.items(): setattr(cfg_pretrained, k, v) kwargs["torch_dtype"] = torch.float16 diff --git a/lmms_eval/models/longva.py b/lmms_eval/models/longva.py index 6f3f3608..ebc7dde8 100644 --- a/lmms_eval/models/longva.py +++ b/lmms_eval/models/longva.py @@ -87,6 +87,7 @@ def __init__( accelerator_kwargs = InitProcessGroupKwargs(timeout=timedelta(weeks=52)) accelerator = Accelerator(kwargs_handlers=[accelerator_kwargs]) + self.accelerator = accelerator if accelerator.num_processes > 1: self._device = torch.device(f"cuda:{accelerator.local_process_index}") self.device_map = f"cuda:{accelerator.local_process_index}" diff --git a/lmms_eval/models/mantis.py b/lmms_eval/models/mantis.py new file mode 100644 index 00000000..7b0a1569 --- /dev/null +++ b/lmms_eval/models/mantis.py @@ -0,0 +1,309 @@ +import torch + +torch.backends.cuda.matmul.allow_tf32 = True + + +import copy +from tqdm import tqdm +from datetime import timedelta + +from lmms_eval import utils +from lmms_eval.api.instance import Instance +from lmms_eval.api.model import lmms +from lmms_eval.api.registry import register_model +from lmms_eval.utils import stop_sequences_criteria + +from accelerate import Accelerator, DistributedType, InitProcessGroupKwargs +from accelerate.state import AcceleratorState +from typing import List, Optional, Union, Tuple +from packaging import version +import warnings + +from loguru import logger as eval_logger + +warnings.filterwarnings("ignore") + +try: + from mantis.models.mllava import LlavaForConditionalGeneration, MLlavaProcessor + from mantis.models.mfuyu import MFuyuForCausalLM, MFuyuProcessor + from mantis.models.conversation import conv_mllava_v1 as default_conv, conv_templates + +except Exception as e: + eval_logger.debug("Mantis is not installed. Please install Mantis to use this model.\nError: %s" % e) + +try: + from transformers import AutoModelForVision2Seq, AutoProcessor +except Exception as e: + eval_logger.debug("Upgrade transformers to use Mantis's idefics model.\nError: %s" % e) + +# inference implementation for attention, can be "sdpa", "eager", "flash_attention_2". Seems FA2 is not effective during inference: https://discuss.huggingface.co/t/flash-attention-has-no-effect-on-inference/73453/5 +# if is_flash_attn_2_available: +# best_fit_attn_implementation = "flash_attention_2" # flash_attn has a bug that says: ERROR Error query and key must have the same dtype in generating + +try: + import flash_attn + + best_fit_attn_implementation = "flash_attention_2" +except ImportError: + best_fit_attn_implementation = "eager" + +DEFAULT_IMAGE_TOKEN = "" + + +@register_model("mantis") +class Mantis(lmms): + """ + Mantis Model + This implementation is adpated from the Llava model from llava.py and the Idefics model from idefics.py + """ + + def __init__( + self, + pretrained: str = "TIGER-Lab/Mantis-8B-siglip-llama3", + truncation: Optional[bool] = True, + device: Optional[str] = "cuda:0", + dtype: Optional[Union[str, torch.dtype]] = "float16", + batch_size: Optional[Union[int, str]] = 1, + attn_implementation=best_fit_attn_implementation, + device_map="cuda:0", + use_cache=True, + truncate_context=False, # whether to truncate the context in generation, set it False for LLaVA-1.6 + **kwargs, + ) -> None: + super().__init__() + # Do not use kwargs for now + assert kwargs == {}, f"Unexpected kwargs: {kwargs}" + + accelerator_kwargs = InitProcessGroupKwargs(timeout=timedelta(weeks=52)) + accelerator = Accelerator(kwargs_handlers=[accelerator_kwargs]) + if accelerator.num_processes > 1: + self._device = torch.device(f"cuda:{accelerator.local_process_index}") + self.device_map = f"cuda:{accelerator.local_process_index}" + elif accelerator.num_processes == 1 and device_map == "auto": + self._device = torch.device(device) + self.device_map = device_map + else: + self._device = torch.device(f"cuda:{accelerator.local_process_index}") + self.device_map = f"cuda:{accelerator.local_process_index}" + + self._is_idefics = "idefics" in pretrained.lower() + if isinstance(dtype, str) and dtype != "auto": + dtype = getattr(torch, dtype) + + # Here we load the "non-idefics" Mantis model. + if not self._is_idefics: + if "fuyu" in pretrained.lower(): + self._processor = MFuyuProcessor.from_pretrained(pretrained) + self._model = MFuyuForCausalLM.from_pretrained(pretrained, device_map=self.device_map, attn_implementation=attn_implementation, torch_dtype=dtype) + else: + self._processor = MLlavaProcessor.from_pretrained(pretrained) + self._model = LlavaForConditionalGeneration.from_pretrained(pretrained, device_map=self.device_map, attn_implementation=attn_implementation, torch_dtype=dtype) + + else: + self._processor = AutoProcessor.from_pretrained(pretrained) + self._model = AutoModelForVision2Seq.from_pretrained(pretrained, device_map=self.device_map, torch_dtype=dtype) + eval_logger.info(f"Using {type(self._model)} to instantiate the Mantis model.") + + self._tokenizer = self._processor.tokenizer + + self._config = self._model.config + self.model.eval() + self.model.tie_weights() + self.truncation = truncation + self.batch_size_per_gpu = int(batch_size) + self.use_cache = use_cache + self.truncate_context = truncate_context + + if accelerator.num_processes > 1: + assert accelerator.distributed_type in [DistributedType.FSDP, DistributedType.MULTI_GPU, DistributedType.DEEPSPEED], "Unsupported distributed type provided. Only DDP and FSDP are supported." + # If you want to use DistributedType.DEEPSPEED, you have to run accelerate config before using the model + # Also, you have to select zero stage 0 (equivalent to DDP) in order to make the prepare model works + # I tried to set different parameters in the kwargs to let default zero 2 stage works, but it didn't work. + if accelerator.distributed_type == DistributedType.DEEPSPEED: + kwargs = { + "train_micro_batch_size_per_gpu": self.batch_size_per_gpu, + "train_batch_size": self.batch_size_per_gpu * accelerator.num_processes, + } + AcceleratorState().deepspeed_plugin.deepspeed_config_process(must_match=True, **kwargs) + eval_logger.info("Detected that you are using DistributedType.DEEPSPEED. Make sure you run `accelerate config` and set zero stage to 0") + + if accelerator.distributed_type == DistributedType.FSDP or accelerator.distributed_type == DistributedType.DEEPSPEED: + self._model = accelerator.prepare(self.model) + else: + self._model = accelerator.prepare_model(self.model, evaluation_mode=True) + self.accelerator = accelerator + if self.accelerator.is_local_main_process: + eval_logger.info(f"Using {accelerator.num_processes} devices with data parallelism") + self._rank = self.accelerator.local_process_index + self._world_size = self.accelerator.num_processes + elif accelerator.num_processes == 1 and device_map == "auto": + eval_logger.info(f"Using {accelerator.num_processes} devices with tensor parallelism") + self._rank = 0 + self._word_size = 1 + else: + eval_logger.info(f"Using single device: {self._device}") + self.model.to(self._device) + self._rank = 0 + self._world_size = 1 + + @property + def config(self): + # return the associated transformers.AutoConfig for the given pretrained model. + return self._config + + @property + def tokenizer(self): + return self._tokenizer + + @property + def model(self): + # returns the model, unwrapping it if using Accelerate + if hasattr(self, "accelerator"): + return self.accelerator.unwrap_model(self._model) + else: + return self._model + + @property + def eot_token_id(self): + # we use EOT because end of *text* is more accurate for what we're doing than end of *sentence* + return self.tokenizer.eos_token_id + + @property + def max_length(self): + return self._max_length + + def pad_sequence(self, input_ids, batch_first, padding_value): + if self.tokenizer.padding_side == "left": + input_ids = [torch.flip(_input_ids, [0]) for _input_ids in input_ids] + input_ids = torch.nn.utils.rnn.pad_sequence(input_ids, batch_first=batch_first, padding_value=padding_value) + if self.tokenizer.padding_side == "left": + input_ids = torch.flip(input_ids, [1]) + return input_ids + + @property + def batch_size(self): + return self.batch_size_per_gpu + + @property + def device(self): + return self._device + + @property + def rank(self): + return self._rank + + @property + def world_size(self): + return self._world_size + + def tok_encode(self, string: str, left_truncate_len=None, add_special_tokens=None) -> List[int]: + """ """ + add_special_tokens = False if add_special_tokens is None else add_special_tokens + encoding = self.tokenizer.encode(string, add_special_tokens=add_special_tokens) + # left-truncate the encoded context to be at most `left_truncate_len` tokens long + if left_truncate_len: + encoding = encoding[-left_truncate_len:] + return encoding + + def tok_decode(self, tokens): + try: + return self.tokenizer.decode(tokens) + except: + return self.tokenizer.decode([tokens]) + + def loglikelihood(self, requests: List[Instance]) -> List[Tuple[float, bool]]: + raise NotImplementedError + + def flatten(self, input): + new_list = [] + for i in input: + for j in i: + new_list.append(j) + return new_list + + def generate_until(self, requests: List[Instance]) -> List[str]: + res = [] + + def _collate(x): + # the negative sign on len(toks) sorts descending - this has a few advantages: + # - time estimates will always be over not underestimates, which is more useful for planning + # - to know the size of a batch when going through the list, you know the first one is always the batch + # padded context length. this is useful to simplify the batching logic and more importantly to make + # automatic adaptive batches much much easier to implement + # - any OOMs will happen right away rather than near the end + toks = self.tok_encode(x[0]) + return -len(toks), x[0] + + # we group requests by their generation_kwargs, + # so that we don't try to execute e.g. greedy sampling and temp=0.8 sampling + # in the same batch. + re_ords = utils.Collator([reg.args for reg in requests], _collate, grouping=True) + chunks = re_ords.get_batched(n=self.batch_size, batch_fn=None) + num_iters = len(requests) // self.batch_size if len(requests) % self.batch_size == 0 else len(requests) // self.batch_size + 1 + pbar = tqdm(total=num_iters, disable=(self.rank != 0), desc="Model Responding") + for chunk in chunks: + contexts, all_gen_kwargs, doc_to_visuals, doc_id, tasks, splits = zip(*chunk) + visuals = [doc_to_visual(self.task_dict[task][split][ids]) for ids, task, split, doc_to_visual in zip(doc_id, tasks, splits, doc_to_visuals)] + + # we assume all gen kwargs in the batch are the same + # this is safe to assume because the `grouper` object ensures it. + gen_kwargs = all_gen_kwargs[0] + + until = gen_kwargs.pop("until", None) + image_aspect_ratio = gen_kwargs.pop("image_aspect_ratio", None) + + if "max_new_tokens" not in gen_kwargs: + gen_kwargs["max_new_tokens"] = 1024 + if "temperature" not in gen_kwargs: + gen_kwargs["temperature"] = 0 + + # prompts_input = contexts[0] + + prompts = [] + for visual, context in zip(visuals, contexts): + if self._is_idefics: + # Follow the idefics implementation: + content = [] + if DEFAULT_IMAGE_TOKEN not in context: + for _ in visual: + content.append({"type": "image"}) + content.append({"type": "text", "text": context}) + message = [{"role": "user", "content": content}] + prompt = self._processor.apply_chat_template(message, add_generation_prompt=True) + prompts.append(prompt) + else: + # We follow the Mantis code base: https://github.com/TIGER-AI-Lab/Mantis/blob/main/mantis/models/mllava/utils.py#L33 to make sure they are consistent + # Users don't need to define chat template as it is done here + if "llama-3" in self._model.language_model.name_or_path.lower(): + conv = conv_templates["llama_3"] + terminators = [self._processor.tokenizer.eos_token_id, self._processor.tokenizer.convert_tokens_to_ids("<|eot_id|>")] + else: + conv = default_conv + terminators = None + + gen_kwargs["eos_token_id"] = terminators + + conv = conv.copy() + conv.append_message(conv.roles[0], context) + conv.append_message(conv.roles[1], "") + prompt = conv.get_prompt() + prompts.append(prompt) + inputs = self._processor(images=visuals, text=prompts, return_tensors="pt", truncation=True) + if "image_patches" in inputs.keys(): + inputs["image_patches"] = inputs["image_patches"][0] # FIXME: Fuyu model would return a list instead of a pytorch tensor. This weird behavior needs fixing. + inputs = {k: v.to(self.device) for k, v in inputs.items()} + + output_ids = self.model.generate(**inputs, **gen_kwargs) + for output_id, input_id in zip(output_ids, inputs["input_ids"]): + generated_id = output_id[len(input_id) :] + generated_text = self.tokenizer.decode(generated_id, skip_special_tokens=True) + + res.append(generated_text) + + # self.cache_hook.add_partial("generate_until", (context, gen_kwargs), text_outputs) + pbar.update(1) + # reorder this group of results back to original unsorted form + res = re_ords.get_original(res) + + pbar.close() + return res diff --git a/lmms_eval/tasks/ai2d/ai2d_lite.yaml b/lmms_eval/tasks/ai2d/ai2d_lite.yaml new file mode 100644 index 00000000..b71abe80 --- /dev/null +++ b/lmms_eval/tasks/ai2d/ai2d_lite.yaml @@ -0,0 +1,56 @@ +dataset_path: lmms-lab/LMMs-Eval-Lite +task: "ai2d_lite" +dataset_kwargs: + token: True +dataset_name: ai2d +test_split: lite +output_type: generate_until +doc_to_visual: !function utils.ai2d_doc_to_visual +doc_to_text: !function utils.ai2d_doc_to_text +doc_to_target: !function utils.ai2d_doc_to_target + +lmms_eval_specific_kwargs: + default: + prompt_format: mcq + pre_prompt: "" + post_prompt: "\nAnswer with the option's letter from the given choices directly." + gpt4v: + prompt_format: mcq + pre_prompt: "" + post_prompt: "\nAbove choices are given in {option}. {content} format.\nPlease answer with the option letter from the given choices directly." + qwen_vl: + prompt_format: qa + pre_prompt: "" + post_prompt: " Answer:" + xcomposer2_4khd: + prompt_format: mcq_xcomposer + pre_prompt: "[UNUSED_TOKEN_146]user\nQuestion: " + post_prompt: "[UNUSED_TOKEN_145]\n[UNUSED_TOKEN_146]assistant\nThe answer is" + +model_specific_target_kwargs: + default: "mcq" + qwen_vl: "qa" + +generation_kwargs: + max_new_tokens: 16 + temperature: 0 + do_sample: False + +filter_list: + - name: "flexible-extract" + filter: + - function: !function utils.MultiChoiceRegexFilter + group_select: 0 + ignore_case: true + ignore_punctuation: true + regex_pattern: "([A-Z])\\." + +metric_list: + - metric: exact_match + aggregation: mean + higher_is_better: true + ignore_case: true + ignore_punctuation: true + +metadata: + - version: 0.0 \ No newline at end of file diff --git a/lmms_eval/tasks/chartqa/chartqa_lite.yaml b/lmms_eval/tasks/chartqa/chartqa_lite.yaml new file mode 100644 index 00000000..96daff5f --- /dev/null +++ b/lmms_eval/tasks/chartqa/chartqa_lite.yaml @@ -0,0 +1,35 @@ +dataset_path: lmms-lab/LMMs-Eval-Lite +dataset_kwargs: + token: True +task: "chartqa_lite" +dataset_name: chartqa +test_split: lite +output_type: generate_until +doc_to_visual: !function utils.chartqa_doc_to_visual +doc_to_text: !function utils.chartqa_doc_to_text +doc_to_target: "answer" +generation_kwargs: + max_new_tokens: 16 + temperature: 0 + do_sample: False +process_results: !function utils.chartqa_process_results +metric_list: + - metric: relaxed_overall + aggregation: mean + higher_is_better: true + - metric: relaxed_human_split + aggregation: mean + higher_is_better: true + - metric: relaxed_augmented_split + aggregation: mean + higher_is_better: true +metadata: + - version: 0.0 +lmms_eval_specific_kwargs: + default: + pre_prompt: "" + post_prompt: "\nAnswer the question with a single word." + qwen_vl: + pre_prompt: "" + post_prompt: " Answer:" + diff --git a/lmms_eval/tasks/coco_cap/coco2017_cap_val_lite.yaml b/lmms_eval/tasks/coco_cap/coco2017_cap_val_lite.yaml new file mode 100644 index 00000000..f04fd514 --- /dev/null +++ b/lmms_eval/tasks/coco_cap/coco2017_cap_val_lite.yaml @@ -0,0 +1,45 @@ +dataset_path: lmms-lab/LMMs-Eval-Lite +dataset_kwargs: + token: True +task: "coco2017_cap_val_lite" +dataset_name: coco2017_cap_val +test_split: lite +output_type: generate_until +doc_to_visual: !function utils.coco_doc_to_visual +doc_to_text: !function utils.coco_doc_to_text +doc_to_target: "answer" +generation_kwargs: + max_new_tokens: 64 + temperature: 0 + top_p: 1.0 + num_beams: 1 + do_sample: false +process_results: !function utils.coco_process_result +# Note that the metric name can be either a registed metric function (such as the case for GQA) or a key name returned by process_results +metric_list: + - metric: coco_Bleu_4 + aggregation : !function utils.coco_bleu4 + higher_is_better : true + - metric: coco_Bleu_3 + aggregation : !function utils.coco_bleu3 + higher_is_better : true + - metric: coco_Bleu_2 + aggregation : !function utils.coco_bleu2 + higher_is_better : true + - metric: coco_Bleu_1 + aggregation : !function utils.coco_bleu1 + higher_is_better : true + - metric: coco_METEOR + aggregation : !function utils.coco_meteor + higher_is_better : true + - metric: coco_ROUGE_L + aggregation : !function utils.coco_rougel + higher_is_better : true + - metric: coco_CIDEr + aggregation : !function utils.coco_cider + higher_is_better : true + #- metric: coco_SPICE + # aggregation : !function utils.coco_spice + # higher_is_better : true +metadata: + - version: 0.0 \ No newline at end of file diff --git a/lmms_eval/tasks/docvqa/docvqa_val_lite.yaml b/lmms_eval/tasks/docvqa/docvqa_val_lite.yaml new file mode 100644 index 00000000..95d065df --- /dev/null +++ b/lmms_eval/tasks/docvqa/docvqa_val_lite.yaml @@ -0,0 +1,26 @@ +task: "docvqa_val_lite" +test_split: lite +metric_list: + - metric: anls + aggregation: mean + higher_is_better: true +dataset_path: lmms-lab/LMMs-Eval-Lite +dataset_name: docvqa_val +dataset_kwargs: + token: True +output_type: generate_until +doc_to_visual: !function utils.docvqa_doc_to_visual +doc_to_text: !function utils.docvqa_doc_to_text +doc_to_target: "answers" +generation_kwargs: + max_new_tokens: 32 + temperature: 0 + do_sample: False +lmms_eval_specific_kwargs: + default: + pre_prompt: "" + post_prompt: "\nAnswer the question using a single word or phrase." + qwen_vl: + pre_prompt: "" + post_prompt: " Answer:" + diff --git a/lmms_eval/tasks/flickr30k/flickr30k_test_lite.yaml b/lmms_eval/tasks/flickr30k/flickr30k_test_lite.yaml new file mode 100644 index 00000000..8d5e85c7 --- /dev/null +++ b/lmms_eval/tasks/flickr30k/flickr30k_test_lite.yaml @@ -0,0 +1,45 @@ +dataset_path: lmms-lab/LMMs-Eval-Lite +dataset_kwargs: + token: True +task : "flickr30k_test_lite" +dataset_name: flickr30k_test +test_split: lite +output_type: generate_until +doc_to_visual: !function utils.flickr_doc_to_visual +doc_to_text: !function utils.flickr_doc_to_text +doc_to_target: "answer" +generation_kwargs: + max_new_tokens: 64 + temperature: 0 + top_p: 1.0 + num_beams: 1 + do_sample: false +process_results: !function utils.flickr_process_result +# Note that the metric name can be either a registed metric function (such as the case for GQA) or a key name returned by process_results +metric_list: + - metric: flickr_Bleu_4 + aggregation : !function utils.flickr_bleu4 + higher_is_better : true + - metric: flickr_Bleu_3 + aggregation : !function utils.flickr_bleu3 + higher_is_better : true + - metric: flickr_Bleu_2 + aggregation : !function utils.flickr_bleu2 + higher_is_better : true + - metric: flickr_Bleu_1 + aggregation : !function utils.flickr_bleu1 + higher_is_better : true + - metric: flickr_METEOR + aggregation : !function utils.flickr_meteor + higher_is_better : true + - metric: flickr_ROUGE_L + aggregation : !function utils.flickr_rougel + higher_is_better : true + - metric: flickr_CIDEr + aggregation : !function utils.flickr_cider + higher_is_better : true + #- metric: flickr_SPICE + # aggregation : !function utils.flickr_spice + # higher_is_better : true +metadata: + - version: 0.0 diff --git a/lmms_eval/tasks/gqa/gqa_lite.yaml b/lmms_eval/tasks/gqa/gqa_lite.yaml new file mode 100644 index 00000000..7f432fc4 --- /dev/null +++ b/lmms_eval/tasks/gqa/gqa_lite.yaml @@ -0,0 +1,32 @@ +dataset_path: lmms-lab/LMMs-Eval-Lite +dataset_name: gqa +dataset_kwargs: + token: True +task: "gqa_lite" +test_split: lite +output_type: generate_until +doc_to_visual: !function utils.gqa_doc_to_visual +doc_to_text: !function utils.gqa_doc_to_text +doc_to_target: "answer" +generation_kwargs: + max_new_tokens: 16 + temperature: 0 + top_p: 1.0 + num_beams: 1 + do_sample: false +metric_list: + - metric: exact_match + aggregation: mean + higher_is_better: true + ignore_case: true + ignore_punctuation: true +metadata: + - version: 0.0 + +lmms_eval_specific_kwargs: + default: + pre_prompt: "" + post_prompt: "\nAnswer the question using a single word or phrase." + qwen_vl: + pre_prompt: "" + post_prompt: " Answer:" \ No newline at end of file diff --git a/lmms_eval/tasks/infovqa/infovqa_val_lite.yaml b/lmms_eval/tasks/infovqa/infovqa_val_lite.yaml new file mode 100644 index 00000000..f52ded6e --- /dev/null +++ b/lmms_eval/tasks/infovqa/infovqa_val_lite.yaml @@ -0,0 +1,22 @@ +task: "infovqa_val_lite" +test_split: lite +output_type: generate_until +metric_list: + - metric: anls + aggregation: mean + higher_is_better: true +dataset_path: lmms-lab/LMMs-Eval-Lite +dataset_name: infovqa_val +dataset_kwargs: + token: True +doc_to_target: "answers" +doc_to_visual: !function utils.infovqa_doc_to_visual +doc_to_text: !function utils.infovqa_doc_to_text +generation_kwargs: + max_new_tokens: 32 + temperature: 0 + do_sample: False +lmms_eval_specific_kwargs: + default: + pre_prompt: "" + post_prompt: "\nAnswer the question using a single word or phrase." \ No newline at end of file diff --git a/lmms_eval/tasks/mirb/mirb.yaml b/lmms_eval/tasks/mirb/mirb.yaml new file mode 100644 index 00000000..099e578c --- /dev/null +++ b/lmms_eval/tasks/mirb/mirb.yaml @@ -0,0 +1,30 @@ + +dataset_path: VLLMs/MIRB-hf +task: mirb +dataset_kwargs: + token: True +test_split: test +output_type: generate_until +doc_to_visual: !function utils.mirb_doc_to_visual +doc_to_text: !function utils.mirb_doc_to_text +doc_to_target: !function utils.mirb_doc_to_target +process_results: !function utils.mirb_process_results + +lmms_eval_specific_kwargs: + default: + pre_prompt: "" + post_prompt: "" + + +generation_kwargs: + max_new_tokens: 64 + temperature: 0 + do_sample: False + +metric_list: + - metric: mirb_score + aggregation: !function utils.mirb_aggregation + higher_is_better: true + +metadata: + - version: 0.0 diff --git a/lmms_eval/tasks/mirb/utils.py b/lmms_eval/tasks/mirb/utils.py new file mode 100644 index 00000000..3e675d39 --- /dev/null +++ b/lmms_eval/tasks/mirb/utils.py @@ -0,0 +1,305 @@ +from lmms_eval.filters.extraction import ExtendedRegexFilter +from lmms_eval.filters.transformation import MapFilter +import re +import numpy as np + + +import logging + +eval_logger = logging.getLogger("lmms-eval") + + +def get_task_instruction(dataset): + if dataset in ["analogy", "attribute", "plot_code", "visual_chain", "sightseeing"]: + instr = "Answer with a single word." + elif dataset in ["codeu", "food", "image_jigsaw"]: + instr = "Answer with the option symbol." + elif dataset in ["arxiv"]: + instr = "Answer with the paper title." + elif dataset in ["count"]: + instr = "Answer with a single number." + elif dataset in ["3d_scene"]: + instr = "The following images are different views of the same 3D scene. Answer with a single number." + + return instr + + +def mirb_doc_to_text(doc, lmms_eval_specific_kwargs=None): + subset, question = doc["subset"], doc["questions"] + task_instruction = get_task_instruction(subset) + post_prompt = lmms_eval_specific_kwargs["post_prompt"] + pre_prompt = lmms_eval_specific_kwargs["pre_prompt"] + return f"{pre_prompt}{task_instruction}{question}{post_prompt}" + + +def mirb_doc_to_visual(doc): + image_list = [image.convert("RGB") for image in doc["image_list"]] + return image_list + + +def mirb_doc_to_target(doc): + return doc["answers"] + + +def extract_numbers(string): + """ + Exact all forms of numbers from a string with regex. + https://github.com/MMMU-Benchmark/MMMU/blob/51ce7f3e829c16bb44bc5445782686b4c3508794/eval/eval_utils.py#L100 + """ + # Pattern for numbers with commas + pattern_commas = r"-?\b\d{1,3}(?:,\d{3})+\b" + # Pattern for scientific notation + pattern_scientific = r"-?\d+(?:\.\d+)?[eE][+-]?\d+" + # Pattern for simple numbers without commas + pattern_simple = r"-?(?:\d+\.\d+|\.\d+|\d+\b)(?![eE][+-]?\d+)(?![,\d])" + + # Extract numbers with commas + numbers_with_commas = re.findall(pattern_commas, string) + # Extract numbers in scientific notation + numbers_scientific = re.findall(pattern_scientific, string) + # Extract simple numbers without commas + numbers_simple = re.findall(pattern_simple, string) + + # Combine all extracted numbersz + all_numbers = numbers_with_commas + numbers_scientific + numbers_simple + return all_numbers + + +def check_is_number(string): + """ + Check if the given string a number. + https://github.com/MMMU-Benchmark/MMMU/blob/51ce7f3e829c16bb44bc5445782686b4c3508794/eval/eval_utils.py#L65 + """ + try: + float(string.replace(",", "")) + return True + except ValueError: + # check if there's comma inside + return False + + +def normalize_str(string): + """ + Normalize the str to lower case and make them float numbers if possible. + https://github.com/MMMU-Benchmark/MMMU/blob/51ce7f3e829c16bb44bc5445782686b4c3508794/eval/eval_utils.py#L76 + """ + # check if characters in the string + + # if number, numerize it. + string = string.strip() + + is_number = check_is_number(string) + + if is_number: + string = string.replace(",", "") + string = float(string) + # leave 2 decimal + string = round(string, 2) + return [string] + else: # it's likely to be a string + # lower it + string = string.lower() + if len(string) == 1: + return [" " + string, string + " "] # avoid trivial matches + return [string] + + +def parse_multi_choice_response(response): + # here, we assume we have a list, in which each element is + # a list of model responses for some particular input/target pair. + # so we process each of these (same input/target response sets) + # independently (and keep them a list.) + filtered_resps = [] + option_letter_regex = re.compile(r"^\s*([A-Z])\.") + match = option_letter_regex.match(response) + if match: + # If a match is found, append the matched letter + filtered_resps = match.group(1) + else: + # If no match, return the original response + filtered_resps = response + return filtered_resps + + +def parse_open_response(response): + """ + Parse the prediction from the generated response. + Return a list of predicted strings or numbers. + https://github.com/MMMU-Benchmark/MMMU/blob/51ce7f3e829c16bb44bc5445782686b4c3508794/eval/eval_utils.py#L122 + """ + + # content = content.strip("\n").strip(".").strip(" ") + def get_key_subresponses(response): + key_responses = [] + response = response.strip().strip(".").lower() + sub_responses = re.split(r"\.\s(?=[A-Z])|\n", response) + indicators_of_keys = [ + "could be ", + "so ", + "is ", + "thus ", + "therefore ", + "final ", + "answer ", + "result ", + ] + key_responses = [] + for index, resp in enumerate(sub_responses): + # if last one, accept it's an equation (the entire response can be just one sentence with equation) + if index == len(sub_responses) - 1: + indicators_of_keys.extend(["="]) + shortest_key_response = None # the shortest response that may contain the answer (tail part of the response) + for indicator in indicators_of_keys: + if indicator in resp: + if not shortest_key_response: + shortest_key_response = resp.split(indicator)[-1].strip() + else: + if len(resp.split(indicator)[-1].strip()) < len(shortest_key_response): + shortest_key_response = resp.split(indicator)[-1].strip() + # key_responses.append(resp.split(indicator)[1].strip()) + + if shortest_key_response: + # and it's not trivial + if shortest_key_response.strip() not in [ + ":", + ",", + ".", + "!", + "?", + ";", + ":", + "'", + ]: + key_responses.append(shortest_key_response) + if len(key_responses) == 0: # did not found any + return [response] + return key_responses + + # pdb.set_trace() + key_responses = get_key_subresponses(response) + + pred_list = key_responses.copy() # keep the original string response + for resp in key_responses: + pred_list.extend(extract_numbers(resp)) + + tmp_pred_list = [] + for i in range(len(pred_list)): + tmp_pred_list.extend(normalize_str(pred_list[i])) + pred_list = tmp_pred_list + + # remove duplicates + pred_list = list(set(pred_list)) + + return pred_list + + +def mirb_process_results(doc, results): + pred = results[0] + subset, answer = doc["subset"], doc["answers"] + if answer in ["A", "B", "C", "D", "E"]: # MCQ tasks + parsed_pred = parse_multi_choice_response(pred) + else: + parsed_pred = parse_open_response(pred) + task_type = doc["subset"] + data_dict = {"question_id": doc["question_id"], "subset": task_type, "pred_answer": parsed_pred, "answers": doc["answers"]} + return {f"mirb_score": data_dict} + + +def eval_multi_choice(gold_i, pred_i): + """ + Evaluate a multiple choice instance. + https://github.com/MMMU-Benchmark/MMMU/blob/51ce7f3e829c16bb44bc5445782686b4c3508794/eval/eval_utils.py#L175 + """ + correct = False + # only they are exactly the same, we consider it as correct + if isinstance(gold_i, list): + for answer in gold_i: + if answer == pred_i: + correct = True + break + else: # gold_i is a string + if gold_i == pred_i: + correct = True + return correct + + +def eval_open(gold_i, pred_i): + """ + Evaluate an open question instance + https://github.com/MMMU-Benchmark/MMMU/blob/51ce7f3e829c16bb44bc5445782686b4c3508794/eval/eval_utils.py#L191 + """ + correct = False + if isinstance(gold_i, list): + # use float to avoid trivial matches + norm_answers = [] + for answer in gold_i: + norm_answers.extend(normalize_str(answer)) + else: + norm_answers = normalize_str(gold_i) + for pred in pred_i: # pred is already normalized in parse response phase + if isinstance(pred, str): # if it's a string, then find if ans in the pred_i + for norm_ans in norm_answers: + # only see if the string answer in the string pred + if isinstance(norm_ans, str) and norm_ans in pred: + if not correct: + correct = True + break + else: # it's a float number + if pred in norm_answers: + if not correct: + correct = True + break + return correct + + +def mirb_aggregation(results): + task_num = {} + score = 0 + task_score = {} + for result in results: + if result["subset"] not in task_score: + task_score[result["subset"]] = 0 + task_num[result["subset"]] = 0 + + if result["answers"] in ["A", "B", "C", "D", "E"]: # MCQ tasks + correct = eval_multi_choice(result["answers"], result["pred_answer"]) + task_score[result["subset"]] += correct + score += correct + else: + correct = eval_open(result["answers"], result["pred_answer"]) + task_score[result["subset"]] += correct + score += correct + task_num[result["subset"]] += 1 + avg_score = score / len(results) + + task_score = {k: v / task_num[k] for k, v in task_score.items()} + + print("Performances for different subsets:") + print("=" * 50) + for k, v in task_score.items(): + print(f"{k} : {v:.2f}") + print("=" * 50) + + # print across evaluation dimension + groups = {"Knowledge": ["food", "sightseeing"], "Reasoning": ["codeu", "plot_code", "analogy", "3d_scene"], "Perception": ["image_jigsaw", "count", "attribute"], "Multi-Hop": ["visual_chain", "arxiv"]} + + # Compute the averages for each group + averages_dict = compute_averages_from_task_scores(task_score, groups) + + # Print group performances + print("Performances for different dimensions:") + print("=" * 50) + for group, score in averages_dict.items(): + print(f"{group} : {score:.2f}") + print("=" * 50) + + return avg_score + + +# Compute averages for each group based on task scores +def compute_averages_from_task_scores(task_score, groups): + averages = {} + for group, features in groups.items(): + values = [task_score[feature] for feature in features] + averages[group] = np.mean(values) + return averages diff --git a/lmms_eval/tasks/mmbench/mmbench_cn_dev_lite.yaml b/lmms_eval/tasks/mmbench/mmbench_cn_dev_lite.yaml new file mode 100644 index 00000000..78b36a65 --- /dev/null +++ b/lmms_eval/tasks/mmbench/mmbench_cn_dev_lite.yaml @@ -0,0 +1,32 @@ +task: "mmbench_cn_dev_lite" +test_split: "lite" +metric_list: + - metric: gpt_eval_score + aggregation: !function cn_utils.mmbench_aggregate_dev_results_eval + higher_is_better: true + - metric: submission + higher_is_better: true + aggregation: !function cn_utils.mmbench_aggregate_dev_results +dataset_path: lmms-lab/LMMs-Eval-Lite +dataset_name: mmbench_cn_dev +dataset_kwargs: + token: True +doc_to_target: "answer" +output_type: generate_until +doc_to_visual: !function cn_utils.mmbench_doc_to_visual +doc_to_text: !function cn_utils.mmbench_doc_to_text +generation_kwargs: + max_new_tokens: 256 + temperature: 0 + top_p: 1.0 + num_beams: 1 + do_sample: false +process_results: !function cn_utils.mmbench_process_results +lmms_eval_specific_kwargs: + default: + pre_prompt: "" + post_prompt: "\nθ―·η›΄ζŽ₯δ½Ώη”¨ζ‰€ζδΎ›ηš„ι€‰ι‘Ήε­—ζ―δ½œδΈΊη­”ζ‘ˆε›žη­”γ€‚" +model_specific_generation_kwargs: + llava: + image_aspect_ratio: original + diff --git a/lmms_eval/tasks/mmbench/mmbench_en_dev_lite.yaml b/lmms_eval/tasks/mmbench/mmbench_en_dev_lite.yaml new file mode 100644 index 00000000..226c4843 --- /dev/null +++ b/lmms_eval/tasks/mmbench/mmbench_en_dev_lite.yaml @@ -0,0 +1,35 @@ +task: "mmbench_en_dev_lite" +test_split: lite +dataset_path: lmms-lab/LMMs-Eval-Lite +dataset_name: mmbench_en_dev +dataset_kwargs: + token: True +doc_to_target: "answer" +lmms_eval_specific_kwargs: + default: + pre_prompt: "" + post_prompt: "\nAnswer with the option's letter from the given choices directly." +doc_to_visual: !function en_utils.mmbench_doc_to_visual +doc_to_text: !function en_utils.mmbench_doc_to_text +doc_to_target: "answer" +process_results: !function en_utils.mmbench_process_results +model_specific_generation_kwargs: + llava: + image_aspect_ratio: original +output_type: generate_until +generation_kwargs: + until: + - "ASSISTANT:" + max_new_tokens: 1024 + temperature: 0 + top_p: 1.0 + num_beams: 1 + do_sample: false + +metric_list: + - metric: gpt_eval_score + aggregation: !function en_utils.mmbench_aggregate_dev_results_eval + higher_is_better: true + - metric: submission + aggregation: !function en_utils.mmbench_aggregate_dev_results_submission + higher_is_better: true diff --git a/lmms_eval/tasks/muirbench/utils.py b/lmms_eval/tasks/muirbench/utils.py index a88b5c84..d1f1a95a 100644 --- a/lmms_eval/tasks/muirbench/utils.py +++ b/lmms_eval/tasks/muirbench/utils.py @@ -62,14 +62,12 @@ def muir_aggregation(results): task_num[result["task"]] += 1 score = score / len(results) - task_score = {k: v / task_num[k] for k, v in task_score.items()} print("=" * 50) for k, v in task_score.items(): print(f"{k} : {v:.2f}") print("=" * 50) - return score diff --git a/lmms_eval/tasks/nocaps/nocaps_val_lite.yaml b/lmms_eval/tasks/nocaps/nocaps_val_lite.yaml new file mode 100644 index 00000000..a297842f --- /dev/null +++ b/lmms_eval/tasks/nocaps/nocaps_val_lite.yaml @@ -0,0 +1,46 @@ +dataset_path: lmms-lab/LMMs-Eval-Lite +dataset_name: nocaps_val +dataset_kwargs: + token: True +task: "nocaps_val_lite" +test_split: lite +output_type: generate_until +doc_to_visual: !function utils.nocaps_doc_to_visual +doc_to_text: !function utils.nocaps_doc_to_text +doc_to_target: "annotations_captions" +generation_kwargs: + max_new_tokens: 64 + temperature: 0 + top_p: 1.0 + num_beams: 1 + do_sample: false +process_results: !function utils.nocaps_process_result +# Note that the metric name can be either a registed metric function (such as the case for GQA) or a key name returned by process_results +metric_list: + - metric: nocaps_Bleu_4 + aggregation : !function utils.nocaps_bleu4 + higher_is_better : true + - metric: nocaps_Bleu_3 + aggregation : !function utils.nocaps_bleu3 + higher_is_better : true + - metric: nocaps_Bleu_2 + aggregation : !function utils.nocaps_bleu2 + higher_is_better : true + - metric: nocaps_Bleu_1 + aggregation : !function utils.nocaps_bleu1 + higher_is_better : true + - metric: nocaps_METEOR + aggregation : !function utils.nocaps_meteor + higher_is_better : true + - metric: nocaps_ROUGE_L + aggregation : !function utils.nocaps_rougel + higher_is_better : true + - metric: nocaps_CIDEr + aggregation : !function utils.nocaps_cider + higher_is_better : true + #- metric: nocaps_SPICE + # aggregation : !function utils.nocaps_spice + # higher_is_better : true +metadata: + - version: 0.0 +include: _default_template_nocaps_yaml \ No newline at end of file diff --git a/lmms_eval/tasks/ok_vqa/ok_vqa_val2014_lite.yaml b/lmms_eval/tasks/ok_vqa/ok_vqa_val2014_lite.yaml new file mode 100644 index 00000000..7567e696 --- /dev/null +++ b/lmms_eval/tasks/ok_vqa/ok_vqa_val2014_lite.yaml @@ -0,0 +1,28 @@ +group: ok_vqa +task: ok_vqa_val2014_lite +test_split: lite +dataset_path: lmms-lab/LMMs-Eval-Lite +dataset_name: ok_vqa_val2014 +output_type: generate_until +doc_to_visual: !function utils.ok_vqa_doc_to_visual +doc_to_text: !function utils.ok_vqa_doc_to_text +doc_to_target: "answer" +generation_kwargs: + until: + - "ASSISTANT:" +metric_list: + - metric: exact_match + aggregation: mean + higher_is_better: true + ignore_case: true + ignore_punctuation: true + - metric: submission + aggregation: !function utils.ok_vqa_aggregate_submissions + higher_is_better: true +process_results: !function utils.ok_vqa_process_results +lmms_eval_specific_kwargs: + default: + pre_prompt: "" + post_prompt: "\nWhen the provided information is insufficient, respond with 'Unanswerable'.\nAnswer the question using a single word or phrase." +metadata: + - version: 0.0 \ No newline at end of file diff --git a/lmms_eval/tasks/refcoco+/_default_template_bbox_rec_yaml b/lmms_eval/tasks/refcoco+/_default_template_bbox_rec_yaml new file mode 100644 index 00000000..c369baee --- /dev/null +++ b/lmms_eval/tasks/refcoco+/_default_template_bbox_rec_yaml @@ -0,0 +1,34 @@ +dataset_path: lmms-lab/RefCOCOPlus +output_type: generate_until +process_docs: !function utils_rec.refcoco_bbox_rec_preprocess_dataset +doc_to_visual: !function utils_rec.refcoco_bbox_rec_doc_to_visual +doc_to_text: !function utils_rec.refcoco_bbox_rec_doc_to_text +doc_to_target: "bbox" +generation_kwargs: + until: + - "ASSISTANT:" +process_results: !function utils_rec.refcoco_bbox_rec_process_result +metric_list: + - metric: refcoco_IoU + aggregation : !function utils_rec.refcoco_bbox_rec_iou + higher_is_better : true + - metric: refcoco_ACC@0.1 + aggregation : !function utils_rec.refcoco_bbox_rec_acc01 + higher_is_better : true + - metric: refcoco_ACC@0.3 + aggregation : !function utils_rec.refcoco_bbox_rec_acc03 + higher_is_better : true + - metric: refcoco_ACC@0.5 + aggregation : !function utils_rec.refcoco_bbox_rec_acc05 + higher_is_better : true + - metric: refcoco_ACC@0.7 + aggregation : !function utils_rec.refcoco_bbox_rec_acc07 + higher_is_better : true + - metric: refcoco_ACC@0.9 + aggregation : !function utils_rec.refcoco_bbox_rec_acc09 + higher_is_better : true + - metric: refcoco_Center_ACC + aggregation : !function utils_rec.refcoco_bbox_rec_center_acc + higher_is_better : true +metadata: + version: '0.0' \ No newline at end of file diff --git a/lmms_eval/tasks/refcoco+/refcoco+_bbox_rec_testA.yaml b/lmms_eval/tasks/refcoco+/refcoco+_bbox_rec_testA.yaml new file mode 100644 index 00000000..0ebb6c0c --- /dev/null +++ b/lmms_eval/tasks/refcoco+/refcoco+_bbox_rec_testA.yaml @@ -0,0 +1,4 @@ +group: refcoco+_bbox_rec +task: refcoco+_bbox_rec_testA +include: _default_template_bbox_rec_yaml +test_split: testA diff --git a/lmms_eval/tasks/refcoco+/refcoco+_bbox_rec_testB.yaml b/lmms_eval/tasks/refcoco+/refcoco+_bbox_rec_testB.yaml new file mode 100644 index 00000000..b347bce6 --- /dev/null +++ b/lmms_eval/tasks/refcoco+/refcoco+_bbox_rec_testB.yaml @@ -0,0 +1,4 @@ +group: refcoco+_bbox_rec +task: refcoco+_bbox_rec_testB +include: _default_template_bbox_rec_yaml +test_split: testB diff --git a/lmms_eval/tasks/refcoco+/refcoco+_bbox_rec_val.yaml b/lmms_eval/tasks/refcoco+/refcoco+_bbox_rec_val.yaml new file mode 100644 index 00000000..890f588b --- /dev/null +++ b/lmms_eval/tasks/refcoco+/refcoco+_bbox_rec_val.yaml @@ -0,0 +1,4 @@ +group: refcoco+_bbox_rec +task: refcoco+_bbox_rec_val +include: _default_template_bbox_rec_yaml +test_split: val diff --git a/lmms_eval/tasks/refcoco+/utils_rec.py b/lmms_eval/tasks/refcoco+/utils_rec.py new file mode 100644 index 00000000..ec91bf9c --- /dev/null +++ b/lmms_eval/tasks/refcoco+/utils_rec.py @@ -0,0 +1,221 @@ +import re +import logging +from datasets import Dataset + +eval_logger = logging.getLogger("lmms-eval") + +COCO_REC_METRICS = ["IoU", "ACC@0.1", "ACC@0.3", "ACC@0.5", "ACC@0.7", "ACC@0.9", "Center_ACC"] + + +def refcoco_bbox_rec_preprocess_dataset(dataset: Dataset): + # PIL image stored in dataset['image'] + # add `image_width` and `image_height` to the dataset + dataset = dataset.map(lambda x: {"image_width": x["image"].width, "image_height": x["image"].height}) + + # Original bbox format (top x, top y, width, height) + # Convert to (top-left x, top-left y, bottom-right x, bottom-right y) + # Normalize the bounding box coordinates to be between 0 and 1 + # using the image width and height + dataset = dataset.map( + lambda x: {"bbox": [x["bbox"][0] / x["image_width"], + x["bbox"][1] / x["image_height"], + (x["bbox"][0] + x["bbox"][2]) / x["image_width"], + (x["bbox"][1] + x["bbox"][3]) / x["image_height"]]} + ) + + # currently, the dataset has `answer` as a list of strings + # each answer should be its own row + # we will explode the dataset to have one row per answer + # duplicate the other columns + def explode_answers(example): + answers = example.pop('answer') + return [{'answer': answer, **example} for answer in answers] + + # Apply the function to each element, collecting the results + exploded_rows = [] + for example in dataset: + exploded_rows.extend(explode_answers(example)) + + # Create a new dataset from the exploded rows + new_dataset = Dataset.from_list(exploded_rows) + print(f"Exploded dataset from {len(dataset)} to {len(new_dataset)} rows") + + return new_dataset + + +def refcoco_bbox_rec_doc_to_visual(doc): + # Image is presented as is + image = doc["image"].convert("RGB") + return [image.convert("RGB")] + + +def refcoco_bbox_rec_doc_to_text(doc): + assert isinstance(doc['answer'], str), "Answer must be a string" + return "Bounding box coordinates are specified in the format (top-left x, top-left y, bottom-right x, bottom-right y). All values are floating point numbers bounded between 0 and 1. Please provide the bounding box coordinate of the region this sentence describes: " + doc['answer'] + + +def parse_float_sequence_within(input_str): + """ + Extract the first sequence of four floating-point numbers within square brackets from a string. + + Args: + input_str (str): A string that may contain a sequence of four floats within square brackets. + + Returns: + list: A list of four floats if the pattern is found, or a list of four zeros if the pattern is not found. + """ + # Define the regex pattern to find the first instance of four floats within square brackets + pattern = r'\[\s*(-?\d+(?:\.\d+)?),\s*(-?\d+(?:\.\d+)?),\s*(-?\d+(?:\.\d+)?),\s*(-?\d+(?:\.\d+)?)\s*\]' + + # Use re.search to find the first match of the pattern in the input string + match = re.search(pattern, input_str) + + # If a match is found, convert the captured groups into a list of floats + if match: + return [float(match.group(i)) for i in range(1, 5)] + + # If the input does not contain the pattern, return the null float sequence + return [0, 0, 0, 0] + + +def refcoco_bbox_rec_process_result(doc, result): + """ + Args: + doc: a instance of the eval dataset + results: [pred] + Returns: + a dictionary with key: metric name, value: metric value + """ + pred = result[0] if len(result) > 0 else "" + pred = parse_float_sequence_within(pred) + ann_id = doc["question_id"] + data_dict = {"answer": doc["answer"], "pred": pred, "ann_id": ann_id, 'bbox': doc['bbox']} + return {f"refcoco_{metric}": data_dict for metric in COCO_REC_METRICS} + + +def compute_iou(box1, box2): + """ + Compute the Intersection over Union (IoU) of two bounding boxes. + + Parameters: + - box1 (list of float): Bounding box [x_min, y_min, x_max, y_max]. + - box2 (list of float): Bounding box [x_min, y_min, x_max, y_max]. + + Returns: + - float: IoU of box1 and box2. + """ + # Determine the coordinates of the intersection rectangle + x_left = max(box1[0], box2[0]) + y_top = max(box1[1], box2[1]) + x_right = min(box1[2], box2[2]) + y_bottom = min(box1[3], box2[3]) + + # Compute the area of intersection + intersection_area = max(0, x_right - x_left) * max(0, y_bottom - y_top) + + # Compute the area of both bounding boxes + box1_area = (box1[2] - box1[0]) * (box1[3] - box1[1]) + box2_area = (box2[2] - box2[0]) * (box2[3] - box2[1]) + + # Compute the area of the union + union_area = box1_area + box2_area - intersection_area + + # Compute the Intersection over Union + iou = intersection_area / union_area + + return iou + + +def compute_accuracy(box1, box2, threshold=0.5): + """ + Compute the accuracy of two bounding boxes based on a specified threshold. + + Parameters: + - box1 (list of float): Bounding box [x_min, y_min, x_max, y_max]. + - box2 (list of float): Bounding box [x_min, y_min, x_max, y_max]. + - threshold (float): Threshold for the IoU to consider the prediction correct. + + Returns: + - float: Accuracy of the prediction based on the IoU threshold. + """ + iou = compute_iou(box1, box2) + return iou >= threshold + + +def compute_center_accuracy(box1, box2): + """ + Compute if the center point of box 2 is within box 1. + + Parameters: + - box1 (list of float): Bounding box [x_min, y_min, x_max, y_max]. + - box2 (list of float): Bounding box [x_min, y_min, x_max, y_max]. + + Returns: + - bool: True if the center point of box 2 is within box 1, False otherwise. + """ + # Compute the center point of box 2 + center_x = (box2[0] + box2[2]) / 2 + center_y = (box2[1] + box2[3]) / 2 + + # Check if the center point is within box 1 + return box1[0] <= center_x <= box1[2] and box1[1] <= center_y <= box1[3] + + +def refcoco_bbox_rec_aggregation_result(results, metric): + """ + Aggregate the results of the RefCOCO evaluation task using the specified metric. + + Args: + - results (list of dict): List of result dictionaries. + - metric (str): Metric to use for aggregation. + + Returns: + - dict: Dictionary containing the aggregated results for the specified metric. + """ + scorers = { + 'IoU': compute_iou, + 'ACC@0.1': lambda x, y: compute_accuracy(x, y, 0.1), + 'ACC@0.3': lambda x, y: compute_accuracy(x, y, 0.3), + 'ACC@0.5': lambda x, y: compute_accuracy(x, y, 0.5), + 'ACC@0.7': lambda x, y: compute_accuracy(x, y, 0.7), + 'ACC@0.9': lambda x, y: compute_accuracy(x, y, 0.9), + 'Center_ACC': compute_center_accuracy + } + results_dict = {metric: []} + for result in results: + # Extract the ground truth and predicted bounding boxes + gt_bbox = result['bbox'] + pred_bbox = result['pred'] + # Compute the specified metric between the ground truth and predicted bounding boxes + score = scorers[metric](gt_bbox, pred_bbox) + results_dict[metric].append(score) + results_dict[metric] = sum(results_dict[metric]) / len(results_dict[metric]) + print(f"Aggregated {metric} score: {results_dict[metric]}") + return results_dict[metric] + + +def refcoco_bbox_rec_iou(results): + return refcoco_bbox_rec_aggregation_result(results, "IoU") + + +def refcoco_bbox_rec_acc01(results): + return refcoco_bbox_rec_aggregation_result(results, "ACC@0.1") + +def refcoco_bbox_rec_acc03(results): + return refcoco_bbox_rec_aggregation_result(results, "ACC@0.3") + + +def refcoco_bbox_rec_acc05(results): + return refcoco_bbox_rec_aggregation_result(results, "ACC@0.5") + + +def refcoco_bbox_rec_acc07(results): + return refcoco_bbox_rec_aggregation_result(results, "ACC@0.7") + + +def refcoco_bbox_rec_acc09(results): + return refcoco_bbox_rec_aggregation_result(results, "ACC@0.9") + + +def refcoco_bbox_rec_center_acc(results): + return refcoco_bbox_rec_aggregation_result(results, "Center_ACC") diff --git a/lmms_eval/tasks/refcoco/_default_template_bbox_rec_yaml b/lmms_eval/tasks/refcoco/_default_template_bbox_rec_yaml new file mode 100644 index 00000000..3b5ca100 --- /dev/null +++ b/lmms_eval/tasks/refcoco/_default_template_bbox_rec_yaml @@ -0,0 +1,34 @@ +dataset_path: lmms-lab/RefCOCO +output_type: generate_until +process_docs: !function utils_rec.refcoco_bbox_rec_preprocess_dataset +doc_to_visual: !function utils_rec.refcoco_bbox_rec_doc_to_visual +doc_to_text: !function utils_rec.refcoco_bbox_rec_doc_to_text +doc_to_target: "bbox" +generation_kwargs: + until: + - "ASSISTANT:" +process_results: !function utils_rec.refcoco_bbox_rec_process_result +metric_list: + - metric: refcoco_IoU + aggregation : !function utils_rec.refcoco_bbox_rec_iou + higher_is_better : true + - metric: refcoco_ACC@0.1 + aggregation : !function utils_rec.refcoco_bbox_rec_acc01 + higher_is_better : true + - metric: refcoco_ACC@0.3 + aggregation : !function utils_rec.refcoco_bbox_rec_acc03 + higher_is_better : true + - metric: refcoco_ACC@0.5 + aggregation : !function utils_rec.refcoco_bbox_rec_acc05 + higher_is_better : true + - metric: refcoco_ACC@0.7 + aggregation : !function utils_rec.refcoco_bbox_rec_acc07 + higher_is_better : true + - metric: refcoco_ACC@0.9 + aggregation : !function utils_rec.refcoco_bbox_rec_acc09 + higher_is_better : true + - metric: refcoco_Center_ACC + aggregation : !function utils_rec.refcoco_bbox_rec_center_acc + higher_is_better : true +metadata: + version: '0.0' \ No newline at end of file diff --git a/lmms_eval/tasks/refcoco/refcoco_bbox_rec_test.yaml b/lmms_eval/tasks/refcoco/refcoco_bbox_rec_test.yaml new file mode 100644 index 00000000..896ed4ac --- /dev/null +++ b/lmms_eval/tasks/refcoco/refcoco_bbox_rec_test.yaml @@ -0,0 +1,4 @@ +group: refcoco_bbox_rec +task: refcoco_bbox_rec_test +test_split: test +include: _default_template_bbox_rec_yaml diff --git a/lmms_eval/tasks/refcoco/refcoco_bbox_rec_testA.yaml b/lmms_eval/tasks/refcoco/refcoco_bbox_rec_testA.yaml new file mode 100644 index 00000000..191268a6 --- /dev/null +++ b/lmms_eval/tasks/refcoco/refcoco_bbox_rec_testA.yaml @@ -0,0 +1,4 @@ +group: refcoco_bbox_rec +task: refcoco_bbox_rec_testA +test_split: testA +include: _default_template_bbox_rec_yaml diff --git a/lmms_eval/tasks/refcoco/refcoco_bbox_rec_testB.yaml b/lmms_eval/tasks/refcoco/refcoco_bbox_rec_testB.yaml new file mode 100644 index 00000000..39b29071 --- /dev/null +++ b/lmms_eval/tasks/refcoco/refcoco_bbox_rec_testB.yaml @@ -0,0 +1,4 @@ +group: refcoco_bbox_rec +task: refcoco_bbox_rec_testB +test_split: testB +include: _default_template_bbox_rec_yaml diff --git a/lmms_eval/tasks/refcoco/refcoco_bbox_rec_val.yaml b/lmms_eval/tasks/refcoco/refcoco_bbox_rec_val.yaml new file mode 100644 index 00000000..f5da6c5e --- /dev/null +++ b/lmms_eval/tasks/refcoco/refcoco_bbox_rec_val.yaml @@ -0,0 +1,4 @@ +group: refcoco_bbox_rec +task: refcoco_bbox_rec_val +test_split: val +include: _default_template_bbox_rec_yaml diff --git a/lmms_eval/tasks/refcoco/refcoco_bbox_val_lite.yaml b/lmms_eval/tasks/refcoco/refcoco_bbox_val_lite.yaml new file mode 100644 index 00000000..aebd9565 --- /dev/null +++ b/lmms_eval/tasks/refcoco/refcoco_bbox_val_lite.yaml @@ -0,0 +1,39 @@ +task: refcoco_bbox_val_lite +test_split: lite +dataset_path: lmms-lab/LMMs-Eval-Lite +dataset_name: refcoco_bbox_val +output_type: generate_until +doc_to_visual: !function utils.refcoco_bbox_doc_to_visual +doc_to_text: !function utils.refcoco_doc_to_text +doc_to_target: "answer" +generation_kwargs: + until: + - "ASSISTANT:" +process_results: !function utils.refcoco_process_result +metric_list: + - metric: refcoco_Bleu_4 + aggregation : !function utils.refcoco_bleu4 + higher_is_better : true + - metric: refcoco_Bleu_3 + aggregation : !function utils.refcoco_bleu3 + higher_is_better : true + - metric: refcoco_Bleu_2 + aggregation : !function utils.refcoco_bleu2 + higher_is_better : true + - metric: refcoco_Bleu_1 + aggregation : !function utils.refcoco_bleu1 + higher_is_better : true + - metric: refcoco_METEOR + aggregation : !function utils.refcoco_meteor + higher_is_better : true + - metric: refcoco_ROUGE_L + aggregation : !function utils.refcoco_rougel + higher_is_better : true + - metric: refcoco_CIDEr + aggregation : !function utils.refcoco_cider + higher_is_better : true + #- metric: refcoco_SPICE + # aggregation : !function utils.refcoco_spice + # higher_is_better : true +metadata: + version: '0.0' diff --git a/lmms_eval/tasks/refcoco/utils_rec.py b/lmms_eval/tasks/refcoco/utils_rec.py new file mode 100644 index 00000000..ec91bf9c --- /dev/null +++ b/lmms_eval/tasks/refcoco/utils_rec.py @@ -0,0 +1,221 @@ +import re +import logging +from datasets import Dataset + +eval_logger = logging.getLogger("lmms-eval") + +COCO_REC_METRICS = ["IoU", "ACC@0.1", "ACC@0.3", "ACC@0.5", "ACC@0.7", "ACC@0.9", "Center_ACC"] + + +def refcoco_bbox_rec_preprocess_dataset(dataset: Dataset): + # PIL image stored in dataset['image'] + # add `image_width` and `image_height` to the dataset + dataset = dataset.map(lambda x: {"image_width": x["image"].width, "image_height": x["image"].height}) + + # Original bbox format (top x, top y, width, height) + # Convert to (top-left x, top-left y, bottom-right x, bottom-right y) + # Normalize the bounding box coordinates to be between 0 and 1 + # using the image width and height + dataset = dataset.map( + lambda x: {"bbox": [x["bbox"][0] / x["image_width"], + x["bbox"][1] / x["image_height"], + (x["bbox"][0] + x["bbox"][2]) / x["image_width"], + (x["bbox"][1] + x["bbox"][3]) / x["image_height"]]} + ) + + # currently, the dataset has `answer` as a list of strings + # each answer should be its own row + # we will explode the dataset to have one row per answer + # duplicate the other columns + def explode_answers(example): + answers = example.pop('answer') + return [{'answer': answer, **example} for answer in answers] + + # Apply the function to each element, collecting the results + exploded_rows = [] + for example in dataset: + exploded_rows.extend(explode_answers(example)) + + # Create a new dataset from the exploded rows + new_dataset = Dataset.from_list(exploded_rows) + print(f"Exploded dataset from {len(dataset)} to {len(new_dataset)} rows") + + return new_dataset + + +def refcoco_bbox_rec_doc_to_visual(doc): + # Image is presented as is + image = doc["image"].convert("RGB") + return [image.convert("RGB")] + + +def refcoco_bbox_rec_doc_to_text(doc): + assert isinstance(doc['answer'], str), "Answer must be a string" + return "Bounding box coordinates are specified in the format (top-left x, top-left y, bottom-right x, bottom-right y). All values are floating point numbers bounded between 0 and 1. Please provide the bounding box coordinate of the region this sentence describes: " + doc['answer'] + + +def parse_float_sequence_within(input_str): + """ + Extract the first sequence of four floating-point numbers within square brackets from a string. + + Args: + input_str (str): A string that may contain a sequence of four floats within square brackets. + + Returns: + list: A list of four floats if the pattern is found, or a list of four zeros if the pattern is not found. + """ + # Define the regex pattern to find the first instance of four floats within square brackets + pattern = r'\[\s*(-?\d+(?:\.\d+)?),\s*(-?\d+(?:\.\d+)?),\s*(-?\d+(?:\.\d+)?),\s*(-?\d+(?:\.\d+)?)\s*\]' + + # Use re.search to find the first match of the pattern in the input string + match = re.search(pattern, input_str) + + # If a match is found, convert the captured groups into a list of floats + if match: + return [float(match.group(i)) for i in range(1, 5)] + + # If the input does not contain the pattern, return the null float sequence + return [0, 0, 0, 0] + + +def refcoco_bbox_rec_process_result(doc, result): + """ + Args: + doc: a instance of the eval dataset + results: [pred] + Returns: + a dictionary with key: metric name, value: metric value + """ + pred = result[0] if len(result) > 0 else "" + pred = parse_float_sequence_within(pred) + ann_id = doc["question_id"] + data_dict = {"answer": doc["answer"], "pred": pred, "ann_id": ann_id, 'bbox': doc['bbox']} + return {f"refcoco_{metric}": data_dict for metric in COCO_REC_METRICS} + + +def compute_iou(box1, box2): + """ + Compute the Intersection over Union (IoU) of two bounding boxes. + + Parameters: + - box1 (list of float): Bounding box [x_min, y_min, x_max, y_max]. + - box2 (list of float): Bounding box [x_min, y_min, x_max, y_max]. + + Returns: + - float: IoU of box1 and box2. + """ + # Determine the coordinates of the intersection rectangle + x_left = max(box1[0], box2[0]) + y_top = max(box1[1], box2[1]) + x_right = min(box1[2], box2[2]) + y_bottom = min(box1[3], box2[3]) + + # Compute the area of intersection + intersection_area = max(0, x_right - x_left) * max(0, y_bottom - y_top) + + # Compute the area of both bounding boxes + box1_area = (box1[2] - box1[0]) * (box1[3] - box1[1]) + box2_area = (box2[2] - box2[0]) * (box2[3] - box2[1]) + + # Compute the area of the union + union_area = box1_area + box2_area - intersection_area + + # Compute the Intersection over Union + iou = intersection_area / union_area + + return iou + + +def compute_accuracy(box1, box2, threshold=0.5): + """ + Compute the accuracy of two bounding boxes based on a specified threshold. + + Parameters: + - box1 (list of float): Bounding box [x_min, y_min, x_max, y_max]. + - box2 (list of float): Bounding box [x_min, y_min, x_max, y_max]. + - threshold (float): Threshold for the IoU to consider the prediction correct. + + Returns: + - float: Accuracy of the prediction based on the IoU threshold. + """ + iou = compute_iou(box1, box2) + return iou >= threshold + + +def compute_center_accuracy(box1, box2): + """ + Compute if the center point of box 2 is within box 1. + + Parameters: + - box1 (list of float): Bounding box [x_min, y_min, x_max, y_max]. + - box2 (list of float): Bounding box [x_min, y_min, x_max, y_max]. + + Returns: + - bool: True if the center point of box 2 is within box 1, False otherwise. + """ + # Compute the center point of box 2 + center_x = (box2[0] + box2[2]) / 2 + center_y = (box2[1] + box2[3]) / 2 + + # Check if the center point is within box 1 + return box1[0] <= center_x <= box1[2] and box1[1] <= center_y <= box1[3] + + +def refcoco_bbox_rec_aggregation_result(results, metric): + """ + Aggregate the results of the RefCOCO evaluation task using the specified metric. + + Args: + - results (list of dict): List of result dictionaries. + - metric (str): Metric to use for aggregation. + + Returns: + - dict: Dictionary containing the aggregated results for the specified metric. + """ + scorers = { + 'IoU': compute_iou, + 'ACC@0.1': lambda x, y: compute_accuracy(x, y, 0.1), + 'ACC@0.3': lambda x, y: compute_accuracy(x, y, 0.3), + 'ACC@0.5': lambda x, y: compute_accuracy(x, y, 0.5), + 'ACC@0.7': lambda x, y: compute_accuracy(x, y, 0.7), + 'ACC@0.9': lambda x, y: compute_accuracy(x, y, 0.9), + 'Center_ACC': compute_center_accuracy + } + results_dict = {metric: []} + for result in results: + # Extract the ground truth and predicted bounding boxes + gt_bbox = result['bbox'] + pred_bbox = result['pred'] + # Compute the specified metric between the ground truth and predicted bounding boxes + score = scorers[metric](gt_bbox, pred_bbox) + results_dict[metric].append(score) + results_dict[metric] = sum(results_dict[metric]) / len(results_dict[metric]) + print(f"Aggregated {metric} score: {results_dict[metric]}") + return results_dict[metric] + + +def refcoco_bbox_rec_iou(results): + return refcoco_bbox_rec_aggregation_result(results, "IoU") + + +def refcoco_bbox_rec_acc01(results): + return refcoco_bbox_rec_aggregation_result(results, "ACC@0.1") + +def refcoco_bbox_rec_acc03(results): + return refcoco_bbox_rec_aggregation_result(results, "ACC@0.3") + + +def refcoco_bbox_rec_acc05(results): + return refcoco_bbox_rec_aggregation_result(results, "ACC@0.5") + + +def refcoco_bbox_rec_acc07(results): + return refcoco_bbox_rec_aggregation_result(results, "ACC@0.7") + + +def refcoco_bbox_rec_acc09(results): + return refcoco_bbox_rec_aggregation_result(results, "ACC@0.9") + + +def refcoco_bbox_rec_center_acc(results): + return refcoco_bbox_rec_aggregation_result(results, "Center_ACC") diff --git a/lmms_eval/tasks/refcocog/_default_template_bbox_rec_yaml b/lmms_eval/tasks/refcocog/_default_template_bbox_rec_yaml new file mode 100644 index 00000000..ce95812b --- /dev/null +++ b/lmms_eval/tasks/refcocog/_default_template_bbox_rec_yaml @@ -0,0 +1,34 @@ +dataset_path: lmms-lab/RefCOCOg +output_type: generate_until +process_docs: !function utils_rec.refcoco_bbox_rec_preprocess_dataset +doc_to_visual: !function utils_rec.refcoco_bbox_rec_doc_to_visual +doc_to_text: !function utils_rec.refcoco_bbox_rec_doc_to_text +doc_to_target: "bbox" +generation_kwargs: + until: + - "ASSISTANT:" +process_results: !function utils_rec.refcoco_bbox_rec_process_result +metric_list: + - metric: refcoco_IoU + aggregation : !function utils_rec.refcoco_bbox_rec_iou + higher_is_better : true + - metric: refcoco_ACC@0.1 + aggregation : !function utils_rec.refcoco_bbox_rec_acc01 + higher_is_better : true + - metric: refcoco_ACC@0.3 + aggregation : !function utils_rec.refcoco_bbox_rec_acc03 + higher_is_better : true + - metric: refcoco_ACC@0.5 + aggregation : !function utils_rec.refcoco_bbox_rec_acc05 + higher_is_better : true + - metric: refcoco_ACC@0.7 + aggregation : !function utils_rec.refcoco_bbox_rec_acc07 + higher_is_better : true + - metric: refcoco_ACC@0.9 + aggregation : !function utils_rec.refcoco_bbox_rec_acc09 + higher_is_better : true + - metric: refcoco_Center_ACC + aggregation : !function utils_rec.refcoco_bbox_rec_center_acc + higher_is_better : true +metadata: + version: '0.0' \ No newline at end of file diff --git a/lmms_eval/tasks/refcocog/refcocog_bbox_rec_test.yaml b/lmms_eval/tasks/refcocog/refcocog_bbox_rec_test.yaml new file mode 100644 index 00000000..2f435979 --- /dev/null +++ b/lmms_eval/tasks/refcocog/refcocog_bbox_rec_test.yaml @@ -0,0 +1,4 @@ +group: refcocog_bbox_rec +task: refcocog_bbox_rec_test +include: _default_template_bbox_rec_yaml +test_split: test diff --git a/lmms_eval/tasks/refcocog/refcocog_bbox_rec_val.yaml b/lmms_eval/tasks/refcocog/refcocog_bbox_rec_val.yaml new file mode 100644 index 00000000..5e19397a --- /dev/null +++ b/lmms_eval/tasks/refcocog/refcocog_bbox_rec_val.yaml @@ -0,0 +1,4 @@ +group: refcocog_bbox_rec +task: refcocog_bbox_rec_val +include: _default_template_bbox_rec_yaml +test_split: val diff --git a/lmms_eval/tasks/refcocog/utils_rec.py b/lmms_eval/tasks/refcocog/utils_rec.py new file mode 100644 index 00000000..ec91bf9c --- /dev/null +++ b/lmms_eval/tasks/refcocog/utils_rec.py @@ -0,0 +1,221 @@ +import re +import logging +from datasets import Dataset + +eval_logger = logging.getLogger("lmms-eval") + +COCO_REC_METRICS = ["IoU", "ACC@0.1", "ACC@0.3", "ACC@0.5", "ACC@0.7", "ACC@0.9", "Center_ACC"] + + +def refcoco_bbox_rec_preprocess_dataset(dataset: Dataset): + # PIL image stored in dataset['image'] + # add `image_width` and `image_height` to the dataset + dataset = dataset.map(lambda x: {"image_width": x["image"].width, "image_height": x["image"].height}) + + # Original bbox format (top x, top y, width, height) + # Convert to (top-left x, top-left y, bottom-right x, bottom-right y) + # Normalize the bounding box coordinates to be between 0 and 1 + # using the image width and height + dataset = dataset.map( + lambda x: {"bbox": [x["bbox"][0] / x["image_width"], + x["bbox"][1] / x["image_height"], + (x["bbox"][0] + x["bbox"][2]) / x["image_width"], + (x["bbox"][1] + x["bbox"][3]) / x["image_height"]]} + ) + + # currently, the dataset has `answer` as a list of strings + # each answer should be its own row + # we will explode the dataset to have one row per answer + # duplicate the other columns + def explode_answers(example): + answers = example.pop('answer') + return [{'answer': answer, **example} for answer in answers] + + # Apply the function to each element, collecting the results + exploded_rows = [] + for example in dataset: + exploded_rows.extend(explode_answers(example)) + + # Create a new dataset from the exploded rows + new_dataset = Dataset.from_list(exploded_rows) + print(f"Exploded dataset from {len(dataset)} to {len(new_dataset)} rows") + + return new_dataset + + +def refcoco_bbox_rec_doc_to_visual(doc): + # Image is presented as is + image = doc["image"].convert("RGB") + return [image.convert("RGB")] + + +def refcoco_bbox_rec_doc_to_text(doc): + assert isinstance(doc['answer'], str), "Answer must be a string" + return "Bounding box coordinates are specified in the format (top-left x, top-left y, bottom-right x, bottom-right y). All values are floating point numbers bounded between 0 and 1. Please provide the bounding box coordinate of the region this sentence describes: " + doc['answer'] + + +def parse_float_sequence_within(input_str): + """ + Extract the first sequence of four floating-point numbers within square brackets from a string. + + Args: + input_str (str): A string that may contain a sequence of four floats within square brackets. + + Returns: + list: A list of four floats if the pattern is found, or a list of four zeros if the pattern is not found. + """ + # Define the regex pattern to find the first instance of four floats within square brackets + pattern = r'\[\s*(-?\d+(?:\.\d+)?),\s*(-?\d+(?:\.\d+)?),\s*(-?\d+(?:\.\d+)?),\s*(-?\d+(?:\.\d+)?)\s*\]' + + # Use re.search to find the first match of the pattern in the input string + match = re.search(pattern, input_str) + + # If a match is found, convert the captured groups into a list of floats + if match: + return [float(match.group(i)) for i in range(1, 5)] + + # If the input does not contain the pattern, return the null float sequence + return [0, 0, 0, 0] + + +def refcoco_bbox_rec_process_result(doc, result): + """ + Args: + doc: a instance of the eval dataset + results: [pred] + Returns: + a dictionary with key: metric name, value: metric value + """ + pred = result[0] if len(result) > 0 else "" + pred = parse_float_sequence_within(pred) + ann_id = doc["question_id"] + data_dict = {"answer": doc["answer"], "pred": pred, "ann_id": ann_id, 'bbox': doc['bbox']} + return {f"refcoco_{metric}": data_dict for metric in COCO_REC_METRICS} + + +def compute_iou(box1, box2): + """ + Compute the Intersection over Union (IoU) of two bounding boxes. + + Parameters: + - box1 (list of float): Bounding box [x_min, y_min, x_max, y_max]. + - box2 (list of float): Bounding box [x_min, y_min, x_max, y_max]. + + Returns: + - float: IoU of box1 and box2. + """ + # Determine the coordinates of the intersection rectangle + x_left = max(box1[0], box2[0]) + y_top = max(box1[1], box2[1]) + x_right = min(box1[2], box2[2]) + y_bottom = min(box1[3], box2[3]) + + # Compute the area of intersection + intersection_area = max(0, x_right - x_left) * max(0, y_bottom - y_top) + + # Compute the area of both bounding boxes + box1_area = (box1[2] - box1[0]) * (box1[3] - box1[1]) + box2_area = (box2[2] - box2[0]) * (box2[3] - box2[1]) + + # Compute the area of the union + union_area = box1_area + box2_area - intersection_area + + # Compute the Intersection over Union + iou = intersection_area / union_area + + return iou + + +def compute_accuracy(box1, box2, threshold=0.5): + """ + Compute the accuracy of two bounding boxes based on a specified threshold. + + Parameters: + - box1 (list of float): Bounding box [x_min, y_min, x_max, y_max]. + - box2 (list of float): Bounding box [x_min, y_min, x_max, y_max]. + - threshold (float): Threshold for the IoU to consider the prediction correct. + + Returns: + - float: Accuracy of the prediction based on the IoU threshold. + """ + iou = compute_iou(box1, box2) + return iou >= threshold + + +def compute_center_accuracy(box1, box2): + """ + Compute if the center point of box 2 is within box 1. + + Parameters: + - box1 (list of float): Bounding box [x_min, y_min, x_max, y_max]. + - box2 (list of float): Bounding box [x_min, y_min, x_max, y_max]. + + Returns: + - bool: True if the center point of box 2 is within box 1, False otherwise. + """ + # Compute the center point of box 2 + center_x = (box2[0] + box2[2]) / 2 + center_y = (box2[1] + box2[3]) / 2 + + # Check if the center point is within box 1 + return box1[0] <= center_x <= box1[2] and box1[1] <= center_y <= box1[3] + + +def refcoco_bbox_rec_aggregation_result(results, metric): + """ + Aggregate the results of the RefCOCO evaluation task using the specified metric. + + Args: + - results (list of dict): List of result dictionaries. + - metric (str): Metric to use for aggregation. + + Returns: + - dict: Dictionary containing the aggregated results for the specified metric. + """ + scorers = { + 'IoU': compute_iou, + 'ACC@0.1': lambda x, y: compute_accuracy(x, y, 0.1), + 'ACC@0.3': lambda x, y: compute_accuracy(x, y, 0.3), + 'ACC@0.5': lambda x, y: compute_accuracy(x, y, 0.5), + 'ACC@0.7': lambda x, y: compute_accuracy(x, y, 0.7), + 'ACC@0.9': lambda x, y: compute_accuracy(x, y, 0.9), + 'Center_ACC': compute_center_accuracy + } + results_dict = {metric: []} + for result in results: + # Extract the ground truth and predicted bounding boxes + gt_bbox = result['bbox'] + pred_bbox = result['pred'] + # Compute the specified metric between the ground truth and predicted bounding boxes + score = scorers[metric](gt_bbox, pred_bbox) + results_dict[metric].append(score) + results_dict[metric] = sum(results_dict[metric]) / len(results_dict[metric]) + print(f"Aggregated {metric} score: {results_dict[metric]}") + return results_dict[metric] + + +def refcoco_bbox_rec_iou(results): + return refcoco_bbox_rec_aggregation_result(results, "IoU") + + +def refcoco_bbox_rec_acc01(results): + return refcoco_bbox_rec_aggregation_result(results, "ACC@0.1") + +def refcoco_bbox_rec_acc03(results): + return refcoco_bbox_rec_aggregation_result(results, "ACC@0.3") + + +def refcoco_bbox_rec_acc05(results): + return refcoco_bbox_rec_aggregation_result(results, "ACC@0.5") + + +def refcoco_bbox_rec_acc07(results): + return refcoco_bbox_rec_aggregation_result(results, "ACC@0.7") + + +def refcoco_bbox_rec_acc09(results): + return refcoco_bbox_rec_aggregation_result(results, "ACC@0.9") + + +def refcoco_bbox_rec_center_acc(results): + return refcoco_bbox_rec_aggregation_result(results, "Center_ACC") diff --git a/lmms_eval/tasks/scienceqa/utils.py b/lmms_eval/tasks/scienceqa/utils.py index 49587b5d..7cd42b71 100755 --- a/lmms_eval/tasks/scienceqa/utils.py +++ b/lmms_eval/tasks/scienceqa/utils.py @@ -33,8 +33,8 @@ def sqa_doc_to_target(doc): def sqa_process_results(doc, results): # I know this is weird, but it's how llava parse it. - target = sqa_doc_to_target(doc) - pred = results[0] + target = sqa_doc_to_target(doc).strip().lower() + pred = results[0].strip().lower() if pred == target: return {"exact_match": 1.0} # pattern: ^[A-Z]\. .* diff --git a/lmms_eval/tasks/seedbench/seedbench_lite.yaml b/lmms_eval/tasks/seedbench/seedbench_lite.yaml new file mode 100644 index 00000000..b8b59851 --- /dev/null +++ b/lmms_eval/tasks/seedbench/seedbench_lite.yaml @@ -0,0 +1,29 @@ +dataset_path: lmms-lab/LMMs-Eval-Lite +dataset_name: seedbench +dataset_kwargs: + token: True +task: "seedbench_lite" +test_split: lite +output_type: generate_until +doc_to_visual: !function utils.seed_doc_to_visual +doc_to_text: !function utils.seed_doc_to_text +doc_to_target: "answer" +generation_kwargs: + until: + - "ASSISTANT:" + image_aspect_ratio: original +# The return value of process_results will be used by metrics +process_results: !function utils.seed_process_result +# Note that the metric name can be either a registed metric function (such as the case for GQA) or a key name returned by process_results +metric_list: + - metric: seed_image + aggregation: !function utils.seed_aggregation_result + higher_is_better: true + - metric: seed_video + aggregation: !function utils.seed_aggregation_result + higher_is_better: true + - metric: seed_all + aggregation: !function utils.seed_aggregation_result + higher_is_better: true +metadata: + - version: 0.0 \ No newline at end of file diff --git a/lmms_eval/tasks/seedbench/utils.py b/lmms_eval/tasks/seedbench/utils.py index c2938f13..bf5b3333 100755 --- a/lmms_eval/tasks/seedbench/utils.py +++ b/lmms_eval/tasks/seedbench/utils.py @@ -28,7 +28,7 @@ def seed_aggregation_result(results): total_count = 0 total_correct = 0 for result in results: - if result["pred"] == result["answer"]: + if result["pred"].lower().strip() == result["answer"].lower().strip(): total_correct += 1 total_count += 1 return total_correct / total_count diff --git a/lmms_eval/tasks/seedbench_2/seedbench_2.yaml b/lmms_eval/tasks/seedbench_2/seedbench_2.yaml index 9f874da3..3b83238c 100755 --- a/lmms_eval/tasks/seedbench_2/seedbench_2.yaml +++ b/lmms_eval/tasks/seedbench_2/seedbench_2.yaml @@ -1,7 +1,7 @@ dataset_path: lmms-lab/SEED-Bench-2 dataset_kwargs: token: True -task: "seedbench-2" +task: "seedbench_2" test_split: test output_type: generate_until doc_to_visual: !function utils.seed_doc_to_visual diff --git a/lmms_eval/tasks/seedbench_2_plus/seedbench_2_plus.yaml b/lmms_eval/tasks/seedbench_2_plus/seedbench_2_plus.yaml new file mode 100755 index 00000000..71620676 --- /dev/null +++ b/lmms_eval/tasks/seedbench_2_plus/seedbench_2_plus.yaml @@ -0,0 +1,43 @@ +dataset_path: doolayer/SEED-Bench-2-Plus +dataset_kwargs: + token: True +task: "seedbench_2_plus" +test_split: test +output_type: generate_until +doc_to_visual: !function utils.seed_doc_to_visual +doc_to_text: !function utils.seed_doc_to_text +doc_to_target: "answer" +generation_kwargs: + until: + - "ASSISTANT:" + max_new_tokens: 16 + image_aspect_ratio: original +# The return value of process_results will be used by metrics +process_results: !function utils.seed_process_result +# Note that the metric name can be either a registed metric function (such as the case for GQA) or a key name returned by process_results +metric_list: + - metric: seedbench_2_plus_Chart + aggregation: !function utils.seed_aggregation_result + higher_is_better: true + - metric: seedbench_2_plus_Map + aggregation: !function utils.seed_aggregation_result + higher_is_better: true + - metric: seedbench_2_plus_Web + aggregation: !function utils.seed_aggregation_result + higher_is_better: true + - metric: seedbench_2_plus_all + aggregation: !function utils.seed_aggregation_result + higher_is_better: true +metadata: + - version: 0.0 + +lmms_eval_specific_kwargs: + llava : + img_token : + post_prompt : "Answer with the option's letter from the given choices directly." + gpt4V : + img_token : + post_prompt : "Answer with the option's letter from the given choices directly." + default : + img_token : + post_prompt : "Answer with the option's letter from the given choices directly." \ No newline at end of file diff --git a/lmms_eval/tasks/seedbench_2_plus/utils.py b/lmms_eval/tasks/seedbench_2_plus/utils.py new file mode 100755 index 00000000..b722d250 --- /dev/null +++ b/lmms_eval/tasks/seedbench_2_plus/utils.py @@ -0,0 +1,54 @@ +import json + + +def seed_doc_to_visual(doc): + return [doc["image"].convert("RGB")] + + +def parse_choice_img(choice: str, img_token: str): + if "jpg" in choice or "png" in choice: + return img_token + return choice + + +def seed_doc_to_text(doc, model_specific_kwargs=None): + question = doc["question"] + question.replace("", model_specific_kwargs["img_token"]) + question += "\n" + f"A. {parse_choice_img(doc['choice_A'], model_specific_kwargs['img_token'])}\n" + question += f"B. {parse_choice_img(doc['choice_B'], model_specific_kwargs['img_token'])}\n" + question += f"C. {parse_choice_img(doc['choice_C'], model_specific_kwargs['img_token'])}\n" + question += f"D. {parse_choice_img(doc['choice_D'], model_specific_kwargs['img_token'])}" + + return f"{question}\n{model_specific_kwargs['post_prompt']}" + + +def seed_process_result(doc, result): + pred = result[0].strip() + if len(pred) > 1: + pred = pred[0] + answer = doc["answer"] + data_type = doc["question_image_type"].capitalize() + + return {f"seedbench_2_plus_{data_type}": {"pred": pred, "answer": answer, "question_id": doc["question_id"]}, f"seedbench_2_plus_all": {"pred": pred, "answer": answer, "question_id": doc["question_id"]}} + + +def seed_aggregation_result(results): + total_count = 0 + total_correct = 0 + for result in results: + if result["pred"].lower().strip() == result["answer"].lower().strip(): + total_correct += 1 + total_count += 1 + return total_correct / total_count if total_count != 0 else 0 + + +def seed_aggregation_result_all(results): + score = seed_aggregation_result(results) + stored_results = [] + for result in results: + stored_results.append({"question_id": result["question_id"], "prediction": result["pred"]}) + with open("./seed_submission.json", "w") as f: + json.dump(stored_results, f, indent=4) + print("Storing files for seed_submission ...") + + return score diff --git a/lmms_eval/tasks/textcaps/textcaps_val_lite.yaml b/lmms_eval/tasks/textcaps/textcaps_val_lite.yaml new file mode 100644 index 00000000..a72a40b6 --- /dev/null +++ b/lmms_eval/tasks/textcaps/textcaps_val_lite.yaml @@ -0,0 +1,48 @@ +dataset_path: lmms-lab/LMMs-Eval-Lite +dataset_name: textcaps_val +dataset_kwargs: + token: True +task: "textcaps_val_lite" +test_split: lite +output_type: generate_until +doc_to_visual: !function utils.textcaps_doc_to_visual +doc_to_text: !function utils.textcaps_doc_to_text +doc_to_target: "answer" +generation_kwargs: + max_new_tokens: 64 + temperature: 0 + top_p: 1.0 + num_beams: 1 + do_sample: false +process_results: !function utils.textcaps_process_result +# Note that the metric name can be either a registed metric function (such as the case for GQA) or a key name returned by process_results +metric_list: + - metric: textcaps_Bleu_4 + aggregation : !function utils.textcaps_bleu4 + higher_is_better : true + - metric: textcaps_Bleu_3 + aggregation : !function utils.textcaps_bleu3 + higher_is_better : true + - metric: textcaps_Bleu_2 + aggregation : !function utils.textcaps_bleu2 + higher_is_better : true + - metric: textcaps_Bleu_1 + aggregation : !function utils.textcaps_bleu1 + higher_is_better : true + - metric: textcaps_METEOR + aggregation : !function utils.textcaps_meteor + higher_is_better : true + - metric: textcaps_ROUGE_L + aggregation : !function utils.textcaps_rougel + higher_is_better : true + - metric: textcaps_CIDEr + aggregation : !function utils.textcaps_cider + higher_is_better : true + #- metric: textcaps_SPICE + # aggregation : !function utils.textcaps_spice + # higher_is_better : true +metadata: + - version: 0.0 +lmms_eval_specific_kwargs: + default: + prompt: Provide a one-sentence caption for the provided image. \ No newline at end of file diff --git a/lmms_eval/tasks/textvqa/textvqa_val_lite.yaml b/lmms_eval/tasks/textvqa/textvqa_val_lite.yaml new file mode 100644 index 00000000..dc18a09f --- /dev/null +++ b/lmms_eval/tasks/textvqa/textvqa_val_lite.yaml @@ -0,0 +1,30 @@ +task: textvqa_val_lite +test_split: lite +metric_list: + - metric: exact_match + aggregation: mean + higher_is_better: true + ignore_case: true + ignore_punctuation: true + - metric: submission + aggregation: !function utils.textvqa_aggregate_submissions + higher_is_better: true +dataset_path: lmms-lab/LMMs-Eval-Lite +dataset_name: textvqa_val +output_type: generate_until +doc_to_visual: !function utils.textvqa_doc_to_visual +doc_to_text: !function utils.textvqa_doc_to_text +doc_to_target: "answer" +generation_kwargs: + until: + - "ASSISTANT:" +process_results: !function utils.textvqa_process_results +lmms_eval_specific_kwargs: + default: + pre_prompt: "" + post_prompt: "\nAnswer the question using a single word or phrase." + ocr: false + qwen_vl: + pre_prompt: "" + post_prompt: " Answer:" + diff --git a/lmms_eval/tasks/vcr_wiki/_default_template_vcr_yaml b/lmms_eval/tasks/vcr_wiki/_default_template_vcr_yaml index cbe474b4..3394fbec 100644 --- a/lmms_eval/tasks/vcr_wiki/_default_template_vcr_yaml +++ b/lmms_eval/tasks/vcr_wiki/_default_template_vcr_yaml @@ -15,4 +15,4 @@ generation_kwargs: # Note that the metric name can be either a registed metric function (such as the case for GQA) or a key name returned by process_results metadata: version: 0.0.1 - load_package: False \ No newline at end of file + load_package: False diff --git a/lmms_eval/tasks/vibe_eval/utils.py b/lmms_eval/tasks/vibe_eval/utils.py index d727b986..77d10386 100644 --- a/lmms_eval/tasks/vibe_eval/utils.py +++ b/lmms_eval/tasks/vibe_eval/utils.py @@ -8,6 +8,14 @@ import yaml +try: + from reka import ChatMessage + from reka.client import Reka +except ImportError: + eval_logger.warning("Reka is not installed, please install it by `pip install reka-api`") + +from loguru import logger as eval_logger + try: from reka import ChatMessage from reka.client import Reka diff --git a/lmms_eval/tasks/videomme/utils.py b/lmms_eval/tasks/videomme/utils.py index 471736c2..2bbea246 100644 --- a/lmms_eval/tasks/videomme/utils.py +++ b/lmms_eval/tasks/videomme/utils.py @@ -176,7 +176,7 @@ def extract_subtitles(video_path, subtitle_path): def videomme_doc_to_visual(doc): cache_dir = os.path.join(base_cache_dir, cache_name) video_path = doc["videoID"] + ".mp4" - video_path = os.path.join(cache_dir, video_path) + video_path = os.path.join(cache_dir, "data", video_path) if os.path.exists(video_path): video_path = video_path elif os.path.exists(video_path.replace("mp4", "MP4")): diff --git a/lmms_eval/tasks/vizwiz_vqa/vizwiz_vqa_val_lite.yaml b/lmms_eval/tasks/vizwiz_vqa/vizwiz_vqa_val_lite.yaml new file mode 100644 index 00000000..915fa7cd --- /dev/null +++ b/lmms_eval/tasks/vizwiz_vqa/vizwiz_vqa_val_lite.yaml @@ -0,0 +1,28 @@ +task: vizwiz_vqa_val_lite +test_split: lite +dataset_path: lmms-lab/LMMs-Eval-Lite +dataset_name: vizwiz_vqa_val +output_type: generate_until +doc_to_visual: !function utils.vizwiz_vqa_doc_to_visual +doc_to_text: !function utils.vizwiz_vqa_doc_to_text +doc_to_target: "answer" +generation_kwargs: + until: + - "ASSISTANT:" +metadata: + - version: 0.0 +lmms_eval_specific_kwargs: + default: + pre_prompt: "" + post_prompt: "\nWhen the provided information is insufficient, respond with 'Unanswerable'.\nAnswer the question using a single word or phrase." +process_results: !function utils.vizwiz_vqa_process_results + +metric_list: + - metric: exact_match + aggregation: mean + higher_is_better: true + ignore_case: true + ignore_punctuation: true + # - metric: submission + # aggregation: !function utils.vizwiz_vqa_aggregate_submissions + # higher_is_better: true \ No newline at end of file diff --git a/lmms_eval/tasks/vqav2/vqav2_val_lite.yaml b/lmms_eval/tasks/vqav2/vqav2_val_lite.yaml new file mode 100644 index 00000000..c0211c62 --- /dev/null +++ b/lmms_eval/tasks/vqav2/vqav2_val_lite.yaml @@ -0,0 +1,25 @@ +task: "vqav2_val_lite" +dataset_path: lmms-lab/LMMs-Eval-Lite +dataset_name: vqav2_val +dataset_kwargs: + token: True +output_type: generate_until +doc_to_visual: !function utils.vqav2_doc_to_visual +doc_to_text: !function utils.vqav2_doc_to_text +doc_to_target: "answer" +generation_kwargs: + max_new_tokens: 16 +metadata: + - version: 0.0 +lmms_eval_specific_kwargs: + default: + pre_prompt: "" + post_prompt: "\nAnswer the question using a single word or phrase." +test_split: lite +metric_list: + - metric: exact_match + aggregation: mean + higher_is_better: true + ignore_case: true + ignore_punctuation: true +process_results: !function utils.vqav2_process_results_val diff --git a/tools/lite/embed.py b/tools/lite/embed.py new file mode 100644 index 00000000..e8a0c30e --- /dev/null +++ b/tools/lite/embed.py @@ -0,0 +1,57 @@ +import embedder +from lmms_eval.utils import simple_parse_args_string +from lmms_eval.tasks import initialize_tasks, include_path, get_task_dict, ConfigurableTask +from lmms_eval.api.registry import ALL_TASKS, GROUP_REGISTRY + +import argparse +import os + +import torch.distributed as dist + + +def rank0_print(*args): + if dist.is_initialized(): + if dist.get_rank() == 0: + print(f"Rank {dist.get_rank()}: ", *args) + else: + print(*args) + + +def parse_arguments(): + parser = argparse.ArgumentParser() + parser.add_argument("--name", type=str) + parser.add_argument("--output_path", type=str) + parser.add_argument("--tasks", type=str, required=False, default="") + parser.add_argument("--data_path", type=str, required=False, default="") + parser.add_argument("--image_folder", type=str, required=False, default="") + parser.add_argument("--embedder_kwargs", type=str, default="") + + return parser.parse_args() + + +if __name__ == "__main__": + args = parse_arguments() + embedder_name = args.name + output_path = args.output_path + if args.tasks.lower().strip() == "all": + initialize_tasks() + for task in list(ALL_TASKS): + if task in GROUP_REGISTRY: + ALL_TASKS.remove(task) + tasks = list(ALL_TASKS) + else: + tasks = args.tasks.split(",") + + cached_idx = [] + for idx in range(len(tasks)): + if os.path.exists(os.path.join(output_path, f"{tasks[idx]}_embed.npy")): + rank0_print(f"Task {tasks[idx]} exists in cache folder, load from cache") + cached_idx.append(idx) + tasks = [tasks[idx] for idx in range(len(tasks)) if idx not in cached_idx] + rank0_print(f"Tasks : {tasks}") + embedder_kwargs = simple_parse_args_string(args.embedder_kwargs) + + embedder_cls = getattr(embedder, embedder_name) + embedder_obj = embedder_cls(name=embedder_name, output_path=output_path, **embedder_kwargs) + for task in tasks: + embedder_obj.embed_task(task) diff --git a/tools/lite/embedder/BaseEmbedder.py b/tools/lite/embedder/BaseEmbedder.py new file mode 100644 index 00000000..88031fa4 --- /dev/null +++ b/tools/lite/embedder/BaseEmbedder.py @@ -0,0 +1,62 @@ +from abc import ABC, abstractmethod +from typing import List, Optional, Union, Dict +import os + +from lmms_eval.tasks import initialize_tasks, include_path, get_task_dict, ConfigurableTask +from lmms_eval.api.registry import ALL_TASKS + +import torch.distributed as dist + + +def rank0_print(*args): + if dist.is_initialized(): + if dist.get_rank() == 0: + print(f"Rank {dist.get_rank()}: ", *args) + else: + print(*args) + + +class BaseEmbedder(ABC): + def __init__(self, name: str, output_path: str) -> None: + super().__init__() + self.name = name + self.output_path = output_path + os.makedirs(self.output_path, exist_ok=True) + initialize_tasks() + + def flatten(self, input): + new_list = [] + for i in input: + for j in i: + new_list.append(j) + return new_list + + # A static method to build requests for lmms_eval tasks + # Pass in task name and return a list of Requests + @staticmethod + def init_task(task: str, ignored_ids: Union[set, List] = None): + task_dict = get_task_dict([task], model_name="llava") + task_obj = task_dict[task] + if type(task_obj) == tuple: + group, task_obj = task_obj + DATASET_PATH = task_obj.DATASET_PATH + DATASET_NAME = None + if task_obj.DATASET_NAME is not None: + DATASET_NAME = task_obj.DATASET_NAME + + docs = task_obj.test_docs() if task_obj.has_test_docs() else task_obj.validation_docs() + split = task_obj.config.test_split if task_obj.has_test_docs() else task_obj.config.validation_split + rank0_print(f"\nTask : {task_obj.config.task}\n - #num : {len(task_obj.test_docs()) if task_obj.has_test_docs() else task_obj.validation_docs()}") + task_obj.build_all_requests() + requests = [] + for instance in task_obj.instances: + reqtype = instance.request_type + contexts, all_gen_kwargs, doc_to_visual, doc_id, task, split = instance.args + if ignored_ids is not None and doc_id in ignored_ids: + continue + requests.append(instance) + return DATASET_PATH, DATASET_NAME, split, requests, task_obj, docs + + @abstractmethod + def embed_task(self, task: str): + return diff --git a/tools/lite/embedder/ClipBgeEmbedder.py b/tools/lite/embedder/ClipBgeEmbedder.py new file mode 100644 index 00000000..11973df4 --- /dev/null +++ b/tools/lite/embedder/ClipBgeEmbedder.py @@ -0,0 +1,92 @@ +from copy import deepcopy +from datetime import timedelta +from typing import Optional, Union, List, Sequence, Dict +import os +import json + +from .BaseEmbedder import BaseEmbedder + +from accelerate import Accelerator, DistributedType, InitProcessGroupKwargs +from accelerate.state import AcceleratorState +from accelerate.utils import gather_object + +import numpy as np +from transformers import CLIPModel, CLIPProcessor +from sentence_transformers import SentenceTransformer +from tqdm import tqdm +import torch +from PIL import Image + + +class ClipBgeEmbedder(BaseEmbedder): + def __init__( + self, + name: str, + output_path: str, + mm_pretrained: str = "openai/clip-vit-large-patch14", + txt_pretrained: str = "BAAI/bge-m3", + device: str = "cuda", + device_map: str = "", + ) -> None: + super().__init__(name, output_path) + self.model = CLIPModel.from_pretrained(mm_pretrained) + self.processor = CLIPProcessor.from_pretrained(mm_pretrained) + self.text_model = SentenceTransformer(txt_pretrained) + accelerator_kwargs = InitProcessGroupKwargs(timeout=timedelta(weeks=52)) + self.accelerator = Accelerator(kwargs_handlers=[accelerator_kwargs]) + if self.accelerator.num_processes > 1 and device_map == "": + self.device = torch.device(f"cuda:{self.accelerator.local_process_index}") + self.device_map = f"cuda:{self.accelerator.local_process_index}" + else: + self.device = torch.device(device) + self.device_map = device_map + + self.model.to(self.device) + self.text_model.to(self.device) + + def embed_task(self, task: str, ignored_ids: Union[set, List] = None): + DATASET_PATH, DATASET_NAME, split, requests, task_obj, self.docs = BaseEmbedder.init_task(task, ignored_ids) + self.accelerator.wait_for_everyone() + with self.accelerator.split_between_processes(requests, apply_padding=False) as requests_split: + results = {"outputs": []} + for req in tqdm(requests_split, disable=not self.accelerator.is_main_process): + contexts, all_gen_kwargs, doc_to_visual, doc_id, task, split = req.args + visuals = [doc_to_visual(self.docs[doc_id])] + visuals = self.flatten(visuals) + + text_embedding = self.text_model.encode([contexts]) + text_embedding = torch.from_numpy(text_embedding).flatten() + + if len(visuals) > 0: + img_inputs = self.processor(images=visuals, return_tensors="pt") + img_inputs = {k: v.to(self.device) for k, v in img_inputs.items()} + + # For multiple images, we take the mean of it + image_embedding = self.model.get_image_features(**img_inputs).mean(dim=0).detach().cpu() + else: + image_embedding = torch.zeros(self.model.config.projection_dim) + + embedding = torch.concat([image_embedding, text_embedding]) + + results["outputs"].append(embedding) + results = [results] + + self.accelerator.wait_for_everyone() + results_gathered = gather_object(results) + if self.accelerator.is_main_process: + outputs = [] + for r in results_gathered: + outputs += r["outputs"] + results_gathered = torch.stack(outputs) + np.save(open(os.path.join(self.output_path, f"{task}_embed.npy"), "wb"), results_gathered) + return results_gathered + + +if __name__ == "__main__": + model = CLIPModel.from_pretrained("openai/clip-vit-large-patch14").to("cuda") + processor = CLIPProcessor.from_pretrained("openai/clip-vit-large-patch14") + text = ["a photo of a cat", "a photo of a dog"] + inputs = processor(text=text, return_tensors="pt") + inputs = {k: v.to("cuda") for k, v in inputs.items()} + outputs = model.get_text_features(**inputs) + print(outputs.mean(dim=0).shape) diff --git a/tools/lite/embedder/__init__.py b/tools/lite/embedder/__init__.py new file mode 100644 index 00000000..c86c10ff --- /dev/null +++ b/tools/lite/embedder/__init__.py @@ -0,0 +1,2 @@ +from .BaseEmbedder import BaseEmbedder +from .ClipBgeEmbedder import ClipBgeEmbedder diff --git a/tools/lite/shrink.py b/tools/lite/shrink.py new file mode 100644 index 00000000..23cba7a4 --- /dev/null +++ b/tools/lite/shrink.py @@ -0,0 +1,53 @@ +import argparse +import importlib +import os +import yaml +import json +from pathlib import Path +import hashlib + +from accelerate import Accelerator +from accelerate.utils import InitProcessGroupKwargs +import numpy as np +import datetime + +from lmms_eval.utils import simple_parse_args_string +import shrinker as shrinker_module + + +AVAILABEL_SHRINKER = {"embed": "Embed_Shrinker"} + + +def parse_arguments(): + parser = argparse.ArgumentParser() + parser.add_argument("--shrinker", type=str, help="The type of shrinker you want to use") + parser.add_argument("--num_items", type=str, help="The number of items you want in your shrinked dataset") + parser.add_argument("--tasks", type=str, help="The task you want to shrink. Separate each task with comma, will be parsed in to list") + parser.add_argument("--push_to_hub", action="store_true", default=False, help="Whether to push the shrinked dataset to hub") + parser.add_argument("--shrinker_kwargs", type=str, help="In args=xxx,args2=xxx format. Will be parsed into dict") + + return parser.parse_args() + + +if __name__ == "__main__": + args = parse_arguments() + shrinker_kwargs = simple_parse_args_string(args.shrinker_kwargs) + shrinker_name = args.shrinker + tasks = args.tasks.split(",") + num_items = args.num_items.split(",") + assert len(num_items) == 1 or len(num_items) == len(tasks), "Either provide one num items for all task or one num item for each task" + if len(num_items) == 1: + num_items = [float(num_items[0])] * len(tasks) + else: + num_items = [float(n) for n in num_items] + push_to_hub = args.push_to_hub + assert len(num_items) == len(tasks) or len(num_items) == 1, "Either pass in one num_items for whole tasks, or pass in num items for each task" + assert shrinker_name in AVAILABEL_SHRINKER, f"Unavailable shrinker {shrinker_name}. You can choose from {AVAILABEL_SHRINKER.keys()}" + + kwargs_handler = InitProcessGroupKwargs(timeout=datetime.timedelta(seconds=60000)) + accelerator = Accelerator(kwargs_handlers=[kwargs_handler]) + for idx, task in enumerate(tasks): + shrinker = getattr(shrinker_module, f"{AVAILABEL_SHRINKER[shrinker_name]}") + shrinker = shrinker(task=task, num_items=num_items[idx], push_to_hub=push_to_hub, name=shrinker_name, **shrinker_kwargs) + shrinker.shrink() + accelerator.wait_for_everyone() diff --git a/tools/lite/shrinker/BaseShrinker.py b/tools/lite/shrinker/BaseShrinker.py new file mode 100644 index 00000000..742650d3 --- /dev/null +++ b/tools/lite/shrinker/BaseShrinker.py @@ -0,0 +1,35 @@ +from abc import ABC, abstractmethod +from glob import glob +import os +import json +import numpy as np +from numpy.polynomial.polynomial import polyfit +import matplotlib.pyplot as plt +from sklearn.metrics import mean_squared_error +from typing import Dict, List, Union +from distutils.dir_util import copy_tree +import random +import torch +import lmms_eval +from lmms_eval.evaluator import evaluate +from lmms_eval.tasks import initialize_tasks, include_path, get_task_dict, ConfigurableTask +from lmms_eval.api.registry import ALL_TASKS +import logging +from lmms_eval.utils import simple_parse_args_string + + +eval_logger = logging.getLogger("lmms-eval") + + +class BaseShrinker(ABC): + def __init__(self, task: str, num_items: Union[int, float], name: str, push_to_hub: bool = True) -> None: + + super().__init__() + self.name = name + self.task = task + self.num_items = float(num_items) + self.push_to_hub = push_to_hub + + @abstractmethod + def shrink(self): + return diff --git a/tools/lite/shrinker/EmbedShrinker.py b/tools/lite/shrinker/EmbedShrinker.py new file mode 100644 index 00000000..0d4f892a --- /dev/null +++ b/tools/lite/shrinker/EmbedShrinker.py @@ -0,0 +1,53 @@ +from .BaseShrinker import BaseShrinker +import sys +from .sampling_methods import AVAILABEL_METHODS +from lmms_eval.tasks import initialize_tasks + +import torch +from typing import List, Dict, Union +import numpy as np +import os + +sys.path.append("..") +from embedder import BaseEmbedder +from shrinker import sampling_methods as sampling_methods_module + + +class Embed_Shrinker(BaseShrinker): + def __init__( + self, + task: str, + num_items: Union[int, float], + name: str, + embed_cache_path: str, + sampling_methods: str, + push_to_hub: bool, + ) -> None: + super().__init__(task, num_items, name, push_to_hub) + self.embed_cache_path = embed_cache_path + initialize_tasks() + self.DATASET_PATH, self.DATASET_NAME, self.split, _, self.task_obj, docs = BaseEmbedder.init_task(task) + assert sampling_methods in AVAILABEL_METHODS, f"Not available sampling methods, Choose from {AVAILABEL_METHODS.keys()}" + self.sampling_methods = getattr(sampling_methods_module, AVAILABEL_METHODS[sampling_methods]) + + def shrink(self): + task_embedding = np.load(open(os.path.join(self.embed_cache_path, f"{self.task}_embed.npy"), "rb")) + task_embedding = torch.from_numpy(task_embedding) + # I know torch.squeeze is safe but numpy reshape sometimes may not + # so I just do it here by converting to torch + if len(task_embedding.shape) == 3: + task_embedding = task_embedding.squeeze(1) + task_embedding = task_embedding.numpy() + self.sampling_methods = self.sampling_methods(X=task_embedding) + # centroids = self.cluster(task_embedding) + if self.num_items < 1.0: + self.num_items = int(task_embedding.shape[0] * self.num_items) + else: + self.num_items = int(self.num_items) + anchor_points = self.sampling_methods.select_batch(N=self.num_items) + dataset = self.task_obj.dataset[self.split] + tiny_dataset = dataset.select(anchor_points) + + if self.push_to_hub: + tiny_dataset.push_to_hub(repo_id=f"lmms-lab/LMMs-Eval-Lite", config_name=self.task, split="lite") + return diff --git a/tools/lite/shrinker/__init__.py b/tools/lite/shrinker/__init__.py new file mode 100644 index 00000000..737538ba --- /dev/null +++ b/tools/lite/shrinker/__init__.py @@ -0,0 +1,2 @@ +from .BaseShrinker import BaseShrinker +from .EmbedShrinker import Embed_Shrinker diff --git a/tools/lite/shrinker/sampling_methods/__init__.py b/tools/lite/shrinker/sampling_methods/__init__.py new file mode 100644 index 00000000..6dad5f78 --- /dev/null +++ b/tools/lite/shrinker/sampling_methods/__init__.py @@ -0,0 +1,24 @@ +# Copyright 2017 Google Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +AVAILABEL_METHODS = { + "kcenter_greedy": "kCenterGreedy", +} + +for name, class_name in AVAILABEL_METHODS.items(): + try: + exec(f"from .{name} import {class_name}") + except Exception as e: + print(f"Error : \n\n{e}\n\n When trying to import {class_name}") diff --git a/tools/lite/shrinker/sampling_methods/kcenter_greedy.py b/tools/lite/shrinker/sampling_methods/kcenter_greedy.py new file mode 100644 index 00000000..69d993fa --- /dev/null +++ b/tools/lite/shrinker/sampling_methods/kcenter_greedy.py @@ -0,0 +1,80 @@ +import numpy as np +from sklearn.metrics import pairwise_distances +from tqdm import tqdm + + +from .sampling_def import SamplingMethod + + +class kCenterGreedy(SamplingMethod): + def __init__(self, X: np.array): + self.X = X + self.flat_X = self.flatten_X() + self.name = "kcenter" + self.features = self.flat_X + self.min_distances = None + self.n_obs = self.X.shape[0] + self.already_selected = None + + def update_distances(self, cluster_centers, only_new=True, reset_dist=False): + """Update min distances given cluster centers. + + Args: + cluster_centers: indices of cluster centers + only_new: only calculate distance for newly selected points and update + min_distances. + rest_dist: whether to reset min_distances. + """ + + if reset_dist: + self.min_distances = None + if only_new: + cluster_centers = [d for d in cluster_centers if d not in self.already_selected] + if cluster_centers: + # Update min_distances for all examples given new cluster center. + x = self.features[cluster_centers] + dist = pairwise_distances(self.features, x, metric="euclidean") + + if self.min_distances is None: + self.min_distances = np.min(dist, axis=1).reshape(-1, 1) + else: + self.min_distances = np.minimum(self.min_distances, dist) + + def select_batch(self, N): + """ + Diversity promoting active learning method that greedily forms a batch + to minimize the maximum distance to a cluster center among all unlabeled + datapoints. + + Args: + model: model with scikit-like API with decision_function implemented + already_selected: index of datapoints already selected + N: batch size + + Returns: + indices of points selected to minimize distance to cluster centers + """ + + print("Using flat_X as features.") + + new_batch = [] + + for _ in tqdm(range(N), desc="K-Center Greedy"): + if self.already_selected is None: + # Initialize centers with a randomly selected datapoint + # ind = np.random.choice(np.arange(self.n_obs)) + ind = 0 # To avoid randomness + self.already_selected = [] + else: + ind = np.argmax(self.min_distances) + # New examples should not be in already selected since those points + # should have min_distance of zero to a cluster center. + assert ind not in self.already_selected + + self.update_distances([ind], only_new=True, reset_dist=False) + new_batch.append(ind) + self.already_selected.append(ind) + print("Maximum distance from cluster centers is %0.2f" % max(self.min_distances)) + + new_batch = np.array(new_batch) + return new_batch diff --git a/tools/lite/shrinker/sampling_methods/sampling_def.py b/tools/lite/shrinker/sampling_methods/sampling_def.py new file mode 100644 index 00000000..b7d8a40c --- /dev/null +++ b/tools/lite/shrinker/sampling_methods/sampling_def.py @@ -0,0 +1,45 @@ +# Copyright 2017 Google Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Abstract class for sampling methods. + +Provides interface to sampling methods that allow same signature +for select_batch. Each subclass implements select_batch_ with the desired +signature for readability. +""" + +import abc +import numpy as np + + +class SamplingMethod(object): + __metaclass__ = abc.ABCMeta + + @abc.abstractmethod + def __init__(self, X, **kwargs): + self.X = X + + def flatten_X(self): + shape = self.X.shape + flat_X = self.X + if len(shape) > 2: + flat_X = np.reshape(self.X, (shape[0], np.product(shape[1:]))) + return flat_X + + @abc.abstractmethod + def select_batch(self): + return + + def to_dict(self): + return None