MMLU STEM
General QA · 2020-09-07
Average accuracy on the STEM-category subjects of MMLU, such as mathematics, physics, chemistry, biology, computer science and engineering. It is a strict subset of MMLU and is harder than the full average, so it is kept separate from whole-benchmark MMLU scores.
Top models (higher is better)
| Model | Score |
|---|---|
| Qwen2.5 72B | 82.7 |
| Qwen2.5 32B | 80.9 |
| Qwen2-72B | 79.6 |
| Qwen2.5 14B | 76.4 |
| Qwen2.5-Coder-32B | 75.1 |
| Qwen2.5-Coder-14B | 73.9 |
| Llama 3 70B | 73.7 |
| Qwen2.5 7B | 72.3 |
| Mixtral 8x22B | 71.7 |
| Gemma 2 27B | 71.0 |
| Qwen2-57B-A14B | 69.8 |
| Qwen2.5 Coder 7B | 67.6 |
| Qwen1.5 32B | 66.5 |
| Gemma 2 9B | 65.1 |
| Qwen2-7B | 64.2 |
| Qwen2.5 3B | 62.5 |
| DeepSeek-Coder-V2-Lite-Base | 58.5 |
| Qwen2.5-Coder-3B | 56.0 |
| Llama 3 8B | 55.3 |
| Qwen2.5 1.5B | 54.8 |
| StarCoder2-15B | 49.2 |
| Qwen2.5 Coder 1.5B | 49.0 |
| Qwen2-1.5B | 42.7 |
| StarCoder2-7B | 39.8 |
| Qwen2.5 0.5B | 39.8 |
| CodeQwen1.5 7B | 39.6 |
| DeepSeek-Coder-Base 33B | 39.5 |
| StarCoder2-3B | 34.9 |
| Qwen2.5 Coder 0.5B | 34.4 |
| DeepSeek-Coder-Base 6.7B | 34.2 |
| Qwen2-0.5B | 27.5 |
| DeepSeek-Coder-Base 1.3B | 24.5 |