Benchmarks
347 sources · 1,258 models × 3,038 benchmarks · 158,912 scoresThe safety competency of SEA-HELM's Tamil score: the SEA-Safeguard task group alone (classifying content as safe or harmful under a 14-category safety policy, including culturally specific cases); SEA-HELM has no Tamil toxicity task, scored by balanced accuracy rescaled against chance and floored at 0, from 8 runs per model with prompts in the language. Diagnostic: it measures safety classification rather than a model's own refusals, and reaches the SEA-HELM score only through the Tamil score.
No signal or difficulty estimate. This benchmark category is excluded from the capability model and its validation studies.
| # | Model | Setting | Score | Source |
|---|---|---|---|---|
| 1 | GPT-5.4 | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 89.4 | |
| 2 | Sonnet 4.6 | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 87.0 | |
| 3 | Inkling-Small | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 86.5 | |
| 4 | Qwen3.5 397B A17B | disabled · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning off (the board labels the row non-reasoning) | 86.5 | |
| 5 | GPT-5.4 Mini | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 86.4 | |
| 6 | Gemma 4 31B IT | disabled · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning off (the board labels the row non-reasoning) | 84.7 | |
| 7 | MiniMax M3 | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 84.6 | |
| 8 | GPT-5.5 | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 84.4 | |
| 9 | Qwen3.5-9B | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 83.9 | |
| 10 | GPT-5.6 Terra | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 83.7 | |
| 11 | GPT-5.6 Sol | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 82.5 | |
| 12 | Qwen3.6 35B A3B | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 81.8 | |
| 13 | Qwen3.5 35B A3B | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 81.7 | |
| 14 | Qwen3 VL 32B Instruct | not supported · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); the model has no reasoning mode | 81.3 | |
| 15 | Gemini 3.1 Pro Preview | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 81.3 | |
| 16 | Qwen3.5-27B | disabled · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning off (the board labels the row non-reasoning) | 80.9 | |
| 17 | Muse Glimmer 30B | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 80.5 | |
| 18 | Gemma 4 26B A4B IT | disabled · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning off (the board labels the row non-reasoning) | 79.6 | |
| 19 | Sonnet 5 | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 78.5 | |
| 20 | GPT-5.6 Luna | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 78.5 | |
| 21 | Haiku 4.5 | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 78.5 | |
| 22 | Mistral Medium 3.5 | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 78.2 | |
| 23 | Qwen3.5 122B A10B | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 78.0 | |
| 24 | Mistral Small 4 | disabled · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning off (the board labels the row non-reasoning) | 78.0 | |
| 25 | Qwen3.6 27B | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 76.9 | |
| 26 | Qwen3-VL-235B-A22B-Instruct | not supported · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); the model has no reasoning mode | 76.9 | |
| 27 | Gemini 3.1 Flash-Lite | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 74.8 | |
| 28 | Command A+ (May 2026, BF16) | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 74.1 | |
| 29 | Opus 4.8 | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 74.1 | |
| 30 | Qwen3 14B | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 69.0 | |
| 31 | Qwen3 32B | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 68.7 | |
| 32 | Qwen3-VL-8B-Instruct | not supported · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); the model has no reasoning mode | 68.0 | |
| 33 | Llama 4 Maverick Instruct | not supported · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); the model has no reasoning mode | 67.9 | |
| 34 | Gemma 4 12B IT | disabled · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning off (the board labels the row non-reasoning) | 67.1 | |
| 35 | Gemini 3.5 Flash | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 66.2 | |
| 36 | Gemini 3.8 Flash | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 64.4 | |
| 37 | Gemma 4 E4B IT | disabled · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning off (the board labels the row non-reasoning) | 64.3 | |
| 38 | Gemma 3 12B IT | not supported · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); the model has no reasoning mode | 63.7 | |
| 39 | GPT-6 Astra | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 63.6 | |
| 40 | GLM-4.7 | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 62.3 | |
| 41 | Gemini 3.6 Flash | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 62.1 | |
| 42 | Qwen3.5-4B | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 60.7 | |
| 43 | gpt-oss-120b | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 57.3 | |
| 44 | Qwen3 8B | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 55.9 | |
| 45 | Gemma 3 4B IT | not supported · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); the model has no reasoning mode | 54.3 | |
| 46 | Nemotron 3 Ultra 550B A55B | disabled · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning off (the board labels the row non-reasoning) | 50.9 | |
| 47 | Llama 4 Scout Instruct | not supported · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); the model has no reasoning mode | 49.4 | |
| 48 | Gemma 4 E2B IT | disabled · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning off (the board labels the row non-reasoning) | 46.8 | |
| 49 | Qwen3 VL 4B Instruct | not supported · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); the model has no reasoning mode | 43.5 | |
| 50 | gpt-oss-20b | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 43.0 | |
| 51 | Qwen3.8-Flash-Next | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 39.5 | |
| 52 | Llama 3.3 70B Instruct | not supported · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); the model has no reasoning mode | 35.6 | |
| 53 | Gemma 3 27B IT | not supported · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); the model has no reasoning mode | 19.8 | |
| 54 | GLM 4.7 Flash | disabled · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning off (the board labels the row non-reasoning) | 16.7 | |
| 55 | Apertus 70B Instruct | not supported · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); the model has no reasoning mode | 16.4 | |
| 56 | Olmo 3.1 32B Instruct | not supported · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); the model has no reasoning mode | 9.4 | |
| 57 | Nemotron 3 Super 120B A12B | disabled · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning off (the board labels the row non-reasoning) | 1.8 | |
| 58 | Apertus 8B Instruct | not supported · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); the model has no reasoning mode | 0.0 | |
| 59 | Llama 3.1 8B Instruct | not supported · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); the model has no reasoning mode | 0.0 | |
| 60 | Llama 3.2 3B Instruct | not supported · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); the model has no reasoning mode | 0.0 | |
| 61 | Nemotron 3.5 Lightning 30B A3B BF16 | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 0.0 | |
| 62 | Nemotron 3 Nano 30B A3B | disabled · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning off (the board labels the row non-reasoning) | 0.0 | |
| 63 | Tiny Aya Global | not supported · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); the model has no reasoning mode | 0.0 |
Benchmarks outside its family whose model rankings track it most reliably (Spearman ρ over at least 10 shared models).