MMMLU (Arabic, Chinese, French, German, Japanese, Korean and Spanish)
General QA · 2024-09-13
A seven-language subset of OpenAI's MMMLU, the professionally translated MMLU test set: Arabic, Chinese, French, German, Japanese, Korean and Spanish, 14,042 four-option questions each and 98,294 in all. The score averages the seven per-language accuracies; every language translates the same questions, so items are not independent across languages.
Top models (higher is better)
| Model | Score |
|---|---|
| LFM2-1.2B | 46.7 |
| Qwen3 1.7B | 46.5 |
| LFM2-700M | 43.3 |
| Llama 3.2 1B Instruct | 38.1 |
| LFM2-350M | 38.0 |
| Gemma 3 1B IT | 34.4 |
| Qwen3 0.6B | 30.8 |