MMLU-Redux
General QA · 2024-06-06
MMLU-Redux is a manually re-annotated subset of MMLU in which questions with wrong ground truth, unclear wording or multiple correct options were corrected or flagged, so accuracy is measured against the corrected labels. Its 1.0 release has 3,000 questions in 30 subjects and MMLU-Redux 2.0 has 5,700 across all 57; release reports rarely name the version, so no item count is declared, and shots and prompt format are recorded per row. It is its own family, not an MMLU fallback, because its item set and labels differ; the ZeroEval-harness run is the separate zeroeval-mmlu-redux. Higher is better.
Top models (higher is better)
| Model | Score |
|---|---|
| Gemini 3 Pro Preview | 95.9 |
| Opus 4.5 | 95.6 |
| Kimi K2.6 | 95.3 |
| Opus 4.6 | 95.2 |
| GPT-5.2 | 95.0 |
| Qwen3.7-Max (source unspecified) | 95.0 |
| o3 | 94.9 |
| Qwen3.5 397B A17B | 94.9 |
| DeepSeek-V4-Pro | 94.8 |
| Opus 4 | 94.6 |
| Kimi K2.5 | 94.5 |
| Qwen3.6 Plus (2026-04-02) | 94.5 |
| Qwen3.7-Plus (source unspecified) | 94.5 |
| DeepSeek-V3.2 | 94.5 |
| Kimi K2 Thinking | 94.4 |
| Gemini 2.5 Pro | 94.4 |
| GLM-5 | 94.4 |
| Qwen3.5 Plus (2026-02-15) | 94.3 |
| GLM-5.1 | 94.3 |
| Qwen3.5-Omni-Plus | 94.2 |
| Qwen3.5 122B A10B | 94.0 |
| Qwen3-235B-A22B | 93.8 |
| Qwen3 235B A22B Thinking 2507 | 93.8 |
| DeepSeek-V3.1 | 93.7 |
| GPT-5 Mini | 93.7 |
| Gemma 4 31B IT | 93.7 |
| Qwen3 VL 235B A22B Thinking | 93.7 |
| Sonnet 4 | 93.6 |
| Qwen3.6 27B | 93.5 |
| DeepSeek-R1-0528 | 93.4 |
| Qwen3.5 35B A3B | 93.3 |
| Qwen3.6 35B A3B | 93.3 |
| Qwen3.5-27B | 93.2 |
| Qwen3 235B A22B Instruct 2507 | 93.1 |
| DeepSeek-R1 | 92.9 |
| O4 Mini | 92.8 |
| Qwen3 Max Thinking (2026-01-23) | 92.8 |
| o1 (2024-12-17) | 92.8 |
| Kimi K2 Instruct 0905 | 92.7 |
| Kimi K2 Instruct | 92.7 |