CMMLU
General QA · 2023-06-15
CMMLU is a Chinese multitask knowledge benchmark of 11,528 four-option multiple-choice questions across 67 subjects, including China-specific topics in humanities, social science, STEM and law. Scores report accuracy; shot count is recorded per row. Higher is better.
Top models (higher is better)
| Model | Score |
|---|---|
| Kimi K2 Base | 90.9 |
| DeepSeek-V4-Pro-Base | 90.8 |
| DeepSeek-V4-Flash-Base | 90.4 |
| Qwen2-72B | 90.1 |
| Qwen2-57B-A14B | 88.5 |
| Qwen1.5 110B | 88.3 |
| DeepSeek-V2 | 84.0 |
| Qwen2-7B | 83.9 |
| Qwen 72B | 83.6 |
| Qwen1.5 72B Chat | 82.9 |
| Qwen1.5 32B | 82.3 |
| DeepSeek-V2-Chat | 81.6 |
| Llama 4 Maverick | 81.2 |
| DeepSeek-Coder-V2-Instruct | 80.9 |
| Qwen1.5 MoE A2.7B | 79.2 |
| Moonlight 16B A3B | 78.2 |
| Qwen1.5 14B | 77.6 |
| Llama 3.1 405B | 73.7 |
| Qwen1.5 7B | 73.1 |
| Qwen 14B | 71.0 |
| DeepSeek LLM 67B Base | 70.8 |
| Llama 3 70B Instruct | 70.7 |
| Qwen2-1.5B | 70.3 |
| DeepSeek LLM 67B Chat | 67.8 |
| Qwen1.5 4B | 66.7 |
| DeepSeek-V2-Lite | 64.3 |
| DeepSeek-Coder-V2-Lite-Instruct | 62.7 |
| DeepSeek-V2-Lite-Chat | 62.5 |
| Qwen 7B | 62.2 |
| Baichuan2-13B-Base | 62.0 |
| Mixtral 8x22B Instruct | 61.0 |
| InternLM 20B | 59.0 |
| Qwen1.5 1.8B | 57.8 |
| Baichuan2-7B-Base | 57.1 |
| Qwen2-0.5B | 55.1 |
| Llama 2 70B | 53.6 |
| InternLM 7B | 51.8 |
| Stable Beluga 2 | 51.8 |
| Llama 3 8B | 50.8 |
| DeepSeekMoE 16B Chat | 49.3 |