Benchmarks
347 sources · 1,258 models × 3,038 benchmarks · 158,912 scoresSEA-HELM's Malay score: the unweighted mean of seven competency scores in Malay (understanding, generation, instruction following, multi-turn chat, culture, knowledge and safety), each the mean of its tasks after rescaling against the random baseline, from 8 runs per model with every prompt in Malay. One of the seven language scores the SEA-HELM score averages.
682nd strongest of 798 fitted benchmarks
Signal-to-noise +2.69 for the SEA-HELM family
Another benchmark represents this family in the fit.
| # | Model | Setting | Score | Source |
|---|---|---|---|---|
| 1 | Qwen3.5 397B A17B | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 76.9 | |
| 2 | Gemma 4 31B IT | disabled · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning off (the board labels the row non-reasoning) | 76.7 | |
| 3 | Gemini 3.1 Pro Preview | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 76.3 | |
| 4 | Qwen3.5 122B A10B | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 76.3 | |
| 5 | GPT-5.6 Sol | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 76.3 | |
| 6 | Qwen3.6 27B | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 76.3 | |
| 7 | Gemma 4 26B A4B IT | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 76.0 | |
| 8 | Gemini 3.1 Flash-Lite | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 75.8 | |
| 9 | Gemini 3.8 Flash | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 75.7 | |
| 10 | Gemini 3.6 Flash | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 75.5 | |
| 11 | GPT-5.5 | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 75.5 | |
| 12 | Gemini 3.5 Flash | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 75.3 | |
| 13 | GPT-5.4 | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 75.3 | |
| 14 | GPT-5.6 Terra | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 75.1 | |
| 15 | Qwen3.6 35B A3B | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 75.1 | |
| 16 | Opus 4.8 | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 75.0 | |
| 17 | Qwen3.8-Flash-Next | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 74.9 | |
| 18 | Qwen3.5-27B | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 74.9 | |
| 19 | Sonnet 4.6 | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 74.6 | |
| 20 | Qwen3.5 35B A3B | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 74.4 | |
| 21 | GPT-6 Astra | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 74.3 | |
| 22 | Muse Glimmer 30B | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 74.1 | |
| 23 | Qwen3-VL-235B-A22B-Instruct | not supported · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); the model has no reasoning mode | 74.1 | |
| 24 | Inkling-Small | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 73.9 | |
| 25 | MiniMax M3 | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 73.8 | |
| 26 | GPT-5.4 Mini | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 73.2 | |
| 27 | GLM-4.7 | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 73.1 | |
| 28 | Nemotron 3 Ultra 550B A55B | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 71.9 | |
| 29 | Sonnet 5 | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 71.8 | |
| 30 | Qwen3 32B | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 71.8 | |
| 31 | GPT-5.6 Luna | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 71.4 | |
| 32 | Qwen3 VL 32B Instruct | not supported · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); the model has no reasoning mode | 71.4 | |
| 33 | Qwen3.5-9B | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 71.1 | |
| 34 | Gemma 4 12B IT | disabled · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning off (the board labels the row non-reasoning) | 70.2 | |
| 35 | Gemma 3 27B IT | not supported · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); the model has no reasoning mode | 70.2 | |
| 36 | Haiku 4.5 | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 70.1 | |
| 37 | gpt-oss-120b | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 69.7 | |
| 38 | Llama 3.3 70B Instruct | not supported · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); the model has no reasoning mode | 69.2 | |
| 39 | Mistral Medium 3.5 | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 68.9 | |
| 40 | Llama 4 Scout Instruct | not supported · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); the model has no reasoning mode | 68.8 | |
| 41 | Nemotron 3 Super 120B A12B | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 68.7 | |
| 42 | Qwen3 14B | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 68.6 | |
| 43 | Gemma 4 E4B IT | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 68.4 | |
| 44 | Qwen3.5-4B | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 68.3 | |
| 45 | Qwen3 8B | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 67.5 | |
| 46 | Qwen3-VL-8B-Instruct | not supported · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); the model has no reasoning mode | 67.0 | |
| 47 | Gemma 3 12B IT | not supported · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); the model has no reasoning mode | 66.4 | |
| 48 | gpt-oss-20b | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 66.1 | |
| 49 | GLM 4.7 Flash | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 65.5 | |
| 50 | Command A+ (May 2026, BF16) | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 64.5 | |
| 51 | Qwen3 VL 4B Instruct | not supported · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); the model has no reasoning mode | 63.7 | |
| 52 | Llama 4 Maverick Instruct | not supported · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); the model has no reasoning mode | 62.3 | |
| 53 | Gemma 4 E2B IT | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 61.5 | |
| 54 | Mistral Small 4 | disabled · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning off (the board labels the row non-reasoning) | 60.7 | |
| 55 | Nemotron 3.5 Lightning 30B A3B BF16 | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 58.8 | |
| 56 | Olmo 3.1 32B Instruct | not supported · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); the model has no reasoning mode | 58.1 | |
| 57 | Nemotron 3 Nano 30B A3B | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 58.0 | |
| 58 | Gemma 3 4B IT | not supported · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); the model has no reasoning mode | 55.2 | |
| 59 | Llama 3.1 8B Instruct | not supported · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); the model has no reasoning mode | 53.1 | |
| 60 | Apertus 70B Instruct | not supported · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); the model has no reasoning mode | 44.8 | |
| 61 | Apertus 8B Instruct | not supported · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); the model has no reasoning mode | 40.6 | |
| 62 | Llama 3.2 3B Instruct | not supported · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); the model has no reasoning mode | 37.7 | |
| 63 | Tiny Aya Global | not supported · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); the model has no reasoning mode | 32.2 |
Benchmarks outside its family whose model rankings track it most reliably (Spearman ρ over at least 10 shared models).