MMMU-Pro Vision
Multimodal · 2024-09-04
The vision-only setting of MMMU-Pro: each question and its ten options are rendered inside a screenshot or photo, and the model receives only the image, so it must read the question visually before answering. Scores are accuracy on the 1,730-question test set.
Top models (higher is better)
| Model | Score |
|---|---|
| GPT-4o Mini | 36.5 |
| Llama 3.2 90B Vision Instruct | 33.8 |
| Phi-4-multimodal-instruct | 27.6 |
| LFM3-Omni-2B | 27.3 |
| Llama 3.2 11B Vision Instruct | 23.7 |
| Claude 3 Haiku | 20.1 |