MuirBench
Multimodal · 2024-06-13
MuirBench evaluates robust multi-image understanding with 2,600 multiple-choice questions over 11,264 images across 12 tasks, pairing each answerable question with an unanswerable variant. Scores report overall accuracy.
Top models (higher is better)
| Model | Score |
|---|---|
| Qwen3 VL 32B Thinking | 80.3 |
| Qwen3 VL 235B A22B Thinking | 80.1 |
| Qwen3 VL 30B A3B Thinking | 77.6 |
| GPT-5 | 77.5 |
| Gemini 2.5 Pro | 77.2 |
| Qwen3 VL 8B Thinking | 76.8 |
| Qwen3 VL 4B Thinking | 75.0 |
| Qwen3.5-4B | 72.8 |
| Qwen3 VL 32B Instruct | 72.8 |
| Gemini 3.5 Flash-Lite | 72.7 |
| Gemini 2.5 Flash | 72.7 |
| Gemma 4 12B IT | 71.9 |
| Qwen2.5-VL-72B-Instruct | 70.7 |
| Sonnet 4 | 68.6 |
| Qwen3 VL 2B Thinking | 68.1 |
| LFM3-VL-2B | 66.4 |
| GPT-5.6 Luna | 66.0 |
| GPT-5 Nano | 65.7 |
| Qwen3.5-2B | 65.5 |
| Qwen3-VL-8B-Instruct | 64.4 |
| Qwen3 VL 4B Instruct | 63.8 |
| Qwen3 VL 30B A3B Instruct | 62.9 |
| LFM2.5-VL-3B | 58.3 |
| GPT-5 Mini | 57.5 |
| Gemma 4 E2B IT | 47.5 |
| Qwen3 VL 2B Instruct | 47.4 |