GSM8K (Open LLM Leaderboard v1)
Math
GSM8K's 1,319 test problems as scored by the Open LLM Leaderboard v1: 5-shot greedy generation under lighteval's port of lm-evaluation-harness, scored by exact match on the final answer. Only runs from lighteval commits first used on or after 2023-12-01 belong here; earlier runs with the same prompts scored lower for every model run on both sides. A fixed harness template is a different measurement from the lab-reported runs in gsm8k, so it forms its own factor group.
Top models (higher is better)
| Model | Score |
|---|---|
| Llama 3 70B Instruct | 85.4 |
| Qwen2-72B | 85.3 |
| Mixtral 8x22B Instruct | 82.0 |
| Qwen1.5 110B | 81.0 |
| Phi-3 Medium 128K Instruct | 80.5 |
| Phi-3 Medium 4K Instruct | 79.4 |
| Smaug-72B-v0.1 | 78.7 |
| Llama 3 70B | 76.9 |
| Phi-3-mini-4k-instruct | 74.5 |
| Mixtral 8x22B | 73.7 |
| Qwen2-7B | 72.9 |
| Yi 1.5 9B Chat | 71.9 |
| Yi 1.5 34B Chat | 71.6 |
| Quyen Pro V0.1 | 71.0 |
| C4AI Command R+ | 70.7 |
| Qwen 72B | 70.4 |
| Nous Hermes 2 Yi 34B | 70.1 |
| Nous Hermes 2 Mixtral 8x7B SFT | 69.6 |
| Phi-3 Mini 128K Instruct | 69.5 |
| Llama 3 8B Instruct | 68.7 |
| Qwen1.5 14B | 67.6 |
| DBRX Instruct | 67.3 |
| Yi 1.5 6B Chat | 67.1 |
| OpenChat 3.5 1210 | 66.0 |
| Qwen1.5 72B | 65.7 |
| Solar 10.7B Instruct V1.0 | 64.7 |
| Tülu 2 DPO 70B | 62.6 |
| DeepSeek LLM 67B Chat | 62.3 |
| Mixtral 8x7B Instruct | 61.1 |
| Qwen1.5 32B | 60.9 |
| DeepSeekMath 7B Base | 59.2 |
| Mixtral 8x7B | 57.6 |
| DeepSeek LLM 67B Base | 56.7 |
| Command R | 56.6 |
| Qwen2-1.5B | 55.8 |
| Phi-2 | 55.0 |
| Llama 2 70B | 54.1 |
| Falcon2-11B | 53.8 |
| Qwen1.5 7B | 53.5 |
| Gemma 7B | 52.8 |