MMLU (Open LLM Leaderboard v1)
General QA
MMLU's 14,042 test questions as scored by the Open LLM Leaderboard v1: 5-shot log-likelihood over the four options under lighteval's port of lm-evaluation-harness. The score is the unweighted mean of the 57 subject accuracies rather than item-level accuracy, so it is a mean of means. A fixed harness template is a different measurement from the lab-reported runs in mmlu, so it forms its own factor group.
Top models (higher is better)
| Model | Score |
|---|---|
| Qwen2-72B | 83.8 |
| Qwen1.5 110B | 80.2 |
| Llama 3 70B Instruct | 80.1 |
| Llama 3 70B | 79.2 |
| Qwen1.5 110B Chat | 78.0 |
| Phi-3 Medium 4K Instruct | 77.8 |
| Mixtral 8x22B | 77.8 |
| Mixtral 8x22B Instruct | 77.8 |
| Qwen1.5 72B Chat | 77.4 |
| Qwen 72B | 77.4 |
| Qwen1.5 72B | 77.2 |
| Smaug-72B-v0.1 | 77.2 |
| Yi 1.5 34B Chat | 77.1 |
| Phi-3 Medium 128K Instruct | 76.7 |
| Nous Hermes 2 Yi 34B | 76.7 |
| Yi-34B | 76.4 |
| C4AI Command R+ | 75.7 |
| Qwen1.5 32B Chat | 75.0 |
| Yi 34B Chat | 74.9 |
| Qwen1.5 32B | 74.3 |
| DBRX Instruct | 73.7 |
| Nous Hermes 2 Mixtral 8x7B SFT | 72.2 |
| DeepSeek LLM 67B Chat | 72.2 |
| Mixtral 8x7B | 71.9 |
| DeepSeek LLM 67B Base | 71.8 |
| Mixtral 8x7B Instruct | 71.4 |
| Yi 1.5 9B Chat | 71.0 |
| Falcon 180B | 70.5 |
| Qwen2-7B | 70.5 |
| Yi-9B | 70.1 |
| Tülu 2 DPO 70B | 69.8 |
| Llama 2 70B | 69.8 |
| Qwen1.5 14B | 69.4 |
| Phi-3-mini-4k-instruct | 69.1 |
| Stable Beluga 2 | 68.8 |
| Phi-3 Mini 128K Instruct | 68.7 |
| Qwen1.5 14B Chat | 68.6 |
| Quyen Pro V0.1 | 68.4 |
| Command R | 68.2 |
| Qwen 14B | 67.7 |