Video-MMMU
Multimodal · 2025-01-23
Video-MMMU measures how well multimodal models acquire and apply knowledge from 300 expert-level lecture videos across six disciplines, with 900 questions spanning perception, comprehension and adaptation. Scores are overall accuracy; the frame budget and prompt are the reporter's own and are recorded in score notes. Higher is better.
Top models (higher is better)
| Model | Score |
|---|---|
| Qwen3.8-Max | 88.7 |
| Gemini 3 Pro Preview | 87.6 |
| Gemini 3 Flash Preview | 86.9 |
| Kimi K2.5 | 86.6 |
| GPT-5.2 | 85.9 |
| Qwen3.7-Plus (source unspecified) | 85.4 |
| Gemini 3.1 Pro Preview | 85.3 |
| Opus 4.6 | 85.2 |
| GPT-5.6 Sol | 85.0 |
| Gemini 3.1 Flash-Lite Preview | 84.8 |
| Qwen3.5 397B A17B | 84.7 |
| GPT-5 | 84.6 |
| Opus 4.5 | 84.4 |
| Qwen3.6 27B | 84.4 |
| Qwen3.6 Plus (2026-04-02) | 84.0 |
| Qwen3.6 35B A3B | 83.7 |
| Gemini 2.5 Pro | 83.6 |
| o3 | 83.3 |
| GPT-5 Mini | 82.5 |
| GPT-5.4 | 82.4 |
| Qwen3.5-27B | 82.3 |
| Qwen3.5 122B A10B | 82.0 |
| Gemma 4 31B IT | 81.6 |
| GPT-5.6 Luna | 81.1 |
| Qwen3.5 35B A3B | 80.4 |
| Qwen3 VL 235B A22B Thinking | 80.0 |
| O4 Mini | 79.4 |
| Gemini 2.5 Flash | 79.2 |
| Qwen3 VL 32B Thinking | 79.0 |
| Qwen3.5-9B | 78.9 |
| Sonnet 4.5 | 77.8 |
| Opus 4.1 | 76.2 |
| Gemma 4 26B A4B IT | 76.0 |
| Opus 4.8 | 75.3 |
| Qwen3 VL 30B A3B Thinking | 75.0 |
| Qwen3-VL-235B-A22B-Instruct | 74.7 |
| Grok 4.1 Fast | 74.6 |
| Qwen3.5-4B | 74.1 |
| Gemini 3.5 Flash-Lite | 73.9 |
| Qwen3 VL 8B Thinking | 72.8 |