How can I evaluate specific MME subtasks

[whyisverysmart]

Hi!

This evaluation framework is awesome.

But for MME benchmark, can I get the scores of some specific subtasks, e.g. ocr, count? Will this be a future enhancement? Otherwise I can only recalculate the scores from the evaluation results...

Thanks and look forward to your suggestions.

[kcz358]

Hi, I think currently it is hard to do so because right now the aggregation can only return a float. Possibly the only possible solution is to log the score directly during aggregation. You can check videomme for examples. But we might check if we can improve it or not

[whyisverysmart]

Hi, I think currently it is hard to do so because right now the aggregation can only return a float. Possibly the only possible solution is to log the score directly during aggregation. You can check videomme for examples. But we might check if we can improve it or not

Thanks a lot! I will try this approach.