The results of InstructBLIP got zero on TextVQA

[kinredon]

Thank all the contributors for this awesome codebase
I have evaluated many models using the codebase, the results are corrected. But when I evaluated the InstructBLIP on TextVQA, the results of InstructBLIP got zero as follows:

the command is below:

CUDA_VISIBLE_DEVICES=3 python3 -m accelerate.commands.launch \
    --num_processes 1 \
    --main_process_port 12380 \
    -m lmms_eval \
    --model instructblip \
    --model_args pretrained=Salesforce/instructblip-vicuna-13b \
    --tasks textvqa_val \
    --batch_size 1 \
    --log_samples \
    --wandb_log_samples \
    --log_samples_suffix instructblip \
    --output_path ./logs/

Why does this happen?

[kcz358]

Because the metric we use is exact_match so it does not perform any parsing. You might need to derive the true score from the log.