Benchmarks
347 sources · 1,258 models × 3,038 benchmarks · 158,912 scoresSEA-HELM's Tamil score: the unweighted mean of eight competency scores in Tamil (understanding, generation, reasoning, instruction following, multi-turn chat, culture, LINDSEA linguistic diagnostics and safety), each the mean of its tasks after rescaling against the random baseline, from 8 runs per model with every prompt in Tamil. One of the seven language scores the SEA-HELM score averages.
682nd strongest of 798 fitted benchmarks
Signal-to-noise +2.69 for the SEA-HELM family
Another benchmark represents this family in the fit.
| # | Model | Setting | Score | Source |
|---|---|---|---|---|
| 1 | GPT-5.6 Sol | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 78.1 | |
| 2 | Gemini 3.1 Pro Preview | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 77.3 | |
| 3 | GPT-5.4 | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 77.2 | |
| 4 | GPT-5.5 | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 77.1 | |
| 5 | Sonnet 4.6 | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 76.9 | |
| 6 | Qwen3.5 397B A17B | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 76.7 | |
| 7 | Gemini 3.8 Flash | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 76.6 | |
| 8 | GPT-6 Astra | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 76.4 | |
| 9 | Gemma 4 31B IT | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 76.4 | |
| 10 | GPT-5.6 Terra | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 76.3 | |
| 11 | Sonnet 5 | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 76.0 | |
| 12 | Qwen3.6 35B A3B | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 75.7 | |
| 13 | Gemma 4 26B A4B IT | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 75.7 | |
| 14 | Gemini 3.6 Flash | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 75.6 | |
| 15 | Qwen3.6 27B | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 75.3 | |
| 16 | Gemini 3.5 Flash | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 75.0 | |
| 17 | Qwen3.5 35B A3B | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 74.8 | |
| 18 | Opus 4.8 | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 74.7 | |
| 19 | Qwen3.5-27B | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 74.7 | |
| 20 | Qwen3.5 122B A10B | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 74.7 | |
| 21 | Gemini 3.1 Flash-Lite | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 74.3 | |
| 22 | GPT-5.4 Mini | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 74.2 | |
| 23 | MiniMax M3 | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 73.7 | |
| 24 | Inkling-Small | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 73.5 | |
| 25 | GPT-5.6 Luna | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 73.0 | |
| 26 | Muse Glimmer 30B | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 72.0 | |
| 27 | Qwen3.5-9B | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 70.9 | |
| 28 | Qwen3-VL-235B-A22B-Instruct | not supported · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); the model has no reasoning mode | 70.9 | |
| 29 | GLM-4.7 | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 70.5 | |
| 30 | Gemma 4 12B IT | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 69.6 | |
| 31 | Qwen3 32B | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 69.2 | |
| 32 | Mistral Medium 3.5 | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 68.8 | |
| 33 | Gemma 4 E4B IT | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 67.1 | |
| 34 | Qwen3.8-Flash-Next | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 67.0 | |
| 35 | Qwen3 VL 32B Instruct | not supported · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); the model has no reasoning mode | 66.7 | |
| 36 | Haiku 4.5 | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 66.3 | |
| 37 | gpt-oss-120b | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 66.1 | |
| 38 | Qwen3 14B | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 65.1 | |
| 39 | Gemma 3 12B IT | not supported · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); the model has no reasoning mode | 63.8 | |
| 40 | Llama 4 Scout Instruct | not supported · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); the model has no reasoning mode | 63.6 | |
| 41 | Qwen3.5-4B | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 63.5 | |
| 42 | gpt-oss-20b | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 63.2 | |
| 43 | Gemma 3 27B IT | not supported · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); the model has no reasoning mode | 62.1 | |
| 44 | Nemotron 3 Ultra 550B A55B | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 60.8 | |
| 45 | Gemma 4 E2B IT | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 59.0 | |
| 46 | Command A+ (May 2026, BF16) | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 59.0 | |
| 47 | Llama 4 Maverick Instruct | not supported · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); the model has no reasoning mode | 58.2 | |
| 48 | Llama 3.3 70B Instruct | not supported · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); the model has no reasoning mode | 55.3 | |
| 49 | Mistral Small 4 | disabled · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning off (the board labels the row non-reasoning) | 55.1 | |
| 50 | Qwen3 8B | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 52.5 | |
| 51 | Qwen3-VL-8B-Instruct | not supported · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); the model has no reasoning mode | 46.7 | |
| 52 | Gemma 3 4B IT | not supported · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); the model has no reasoning mode | 43.8 | |
| 53 | GLM 4.7 Flash | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 38.4 | |
| 54 | Qwen3 VL 4B Instruct | not supported · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); the model has no reasoning mode | 35.8 | |
| 55 | Apertus 70B Instruct | not supported · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); the model has no reasoning mode | 27.6 | |
| 56 | Nemotron 3 Super 120B A12B | disabled · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning off (the board labels the row non-reasoning) | 26.8 | |
| 57 | Olmo 3.1 32B Instruct | not supported · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); the model has no reasoning mode | 24.2 | |
| 58 | Nemotron 3 Nano 30B A3B | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 22.5 | |
| 59 | Nemotron 3.5 Lightning 30B A3B BF16 | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 17.2 | |
| 60 | Apertus 8B Instruct | not supported · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); the model has no reasoning mode | 16.7 | |
| 61 | Llama 3.1 8B Instruct | not supported · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); the model has no reasoning mode | 16.4 | |
| 62 | Llama 3.2 3B Instruct | not supported · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); the model has no reasoning mode | 12.8 | |
| 63 | Tiny Aya Global | not supported · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); the model has no reasoning mode | 4.2 |
Benchmarks outside its family whose model rankings track it most reliably (Spearman ρ over at least 10 shared models).