Benchmarks
347 sources · 1,258 models × 3,038 benchmarks · 158,912 scoresThe safety competency of SEA-HELM's Malay score: the mean of toxicity detection (classifying Malay text as clean, offensive or hateful) and the SEA-Safeguard task group (classifying content as safe or harmful under a 14-category safety policy, including culturally specific cases), scored by balanced accuracy rescaled against chance and floored at 0, from 8 runs per model with prompts in the language. Diagnostic: it measures safety classification rather than a model's own refusals, and reaches the SEA-HELM score only through the Malay score.
No signal or difficulty estimate. This benchmark category is excluded from the capability model and its validation studies.
| # | Model | Setting | Score | Source |
|---|---|---|---|---|
| 1 | Gemma 3 27B IT | not supported · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); the model has no reasoning mode | 45.8 | |
| 2 | Gemma 4 31B IT | disabled · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning off (the board labels the row non-reasoning) | 45.8 | |
| 3 | Sonnet 4.6 | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 45.6 | |
| 4 | Llama 4 Maverick Instruct | not supported · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); the model has no reasoning mode | 45.3 | |
| 5 | Muse Glimmer 30B | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 45.3 | |
| 6 | Qwen3.5 397B A17B | disabled · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning off (the board labels the row non-reasoning) | 44.5 | |
| 7 | Qwen3.5 122B A10B | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 43.6 | |
| 8 | Qwen3-VL-235B-A22B-Instruct | not supported · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); the model has no reasoning mode | 43.5 | |
| 9 | Inkling-Small | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 43.4 | |
| 10 | GLM-4.7 | disabled · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning off (the board labels the row non-reasoning) | 43.0 | |
| 11 | Mistral Medium 3.5 | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 43.0 | |
| 12 | Qwen3 32B | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 42.8 | |
| 13 | Qwen3 VL 32B Instruct | not supported · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); the model has no reasoning mode | 42.5 | |
| 14 | Qwen3.5 35B A3B | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 42.4 | |
| 15 | Qwen3.5-27B | disabled · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning off (the board labels the row non-reasoning) | 42.0 | |
| 16 | Gemma 3 12B IT | not supported · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); the model has no reasoning mode | 42.0 | |
| 17 | Gemma 4 26B A4B IT | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 41.4 | |
| 18 | Qwen3.8-Flash-Next | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 41.2 | |
| 19 | Qwen3.5-9B | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 41.0 | |
| 20 | Gemini 3.1 Flash-Lite | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 41.0 | |
| 21 | GPT-5.4 | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 40.9 | |
| 22 | GLM 4.7 Flash | disabled · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning off (the board labels the row non-reasoning) | 40.9 | |
| 23 | MiniMax M3 | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 40.9 | |
| 24 | Gemini 3.1 Pro Preview | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 40.8 | |
| 25 | Qwen3.5-4B | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 40.8 | |
| 26 | Llama 3.3 70B Instruct | not supported · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); the model has no reasoning mode | 40.7 | |
| 27 | Nemotron 3 Ultra 550B A55B | disabled · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning off (the board labels the row non-reasoning) | 40.6 | |
| 28 | Haiku 4.5 | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 40.6 | |
| 29 | Gemma 3 4B IT | not supported · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); the model has no reasoning mode | 39.9 | |
| 30 | Gemma 4 E4B IT | disabled · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning off (the board labels the row non-reasoning) | 39.9 | |
| 31 | Opus 4.8 | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 39.8 | |
| 32 | Qwen3 VL 4B Instruct | not supported · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); the model has no reasoning mode | 39.6 | |
| 33 | Llama 4 Scout Instruct | not supported · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); the model has no reasoning mode | 39.5 | |
| 34 | Qwen3.6 35B A3B | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 39.4 | |
| 35 | Qwen3.6 27B | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 39.3 | |
| 36 | Gemini 3.5 Flash | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 38.7 | |
| 37 | GPT-5.4 Mini | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 38.5 | |
| 38 | Mistral Small 4 | disabled · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning off (the board labels the row non-reasoning) | 38.0 | |
| 39 | GPT-5.5 | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 37.9 | |
| 40 | Qwen3-VL-8B-Instruct | not supported · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); the model has no reasoning mode | 37.8 | |
| 41 | GPT-5.6 Sol | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 37.5 | |
| 42 | GPT-5.6 Terra | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 37.0 | |
| 43 | GPT-5.6 Luna | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 36.8 | |
| 44 | Qwen3 14B | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 36.7 | |
| 45 | Sonnet 5 | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 36.1 | |
| 46 | Gemini 3.8 Flash | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 35.9 | |
| 47 | Nemotron 3 Super 120B A12B | disabled · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning off (the board labels the row non-reasoning) | 35.4 | |
| 48 | Qwen3 8B | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 35.1 | |
| 49 | Gemma 4 12B IT | disabled · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning off (the board labels the row non-reasoning) | 35.0 | |
| 50 | Gemini 3.6 Flash | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 34.7 | |
| 51 | gpt-oss-120b | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 34.4 | |
| 52 | Apertus 70B Instruct | not supported · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); the model has no reasoning mode | 33.7 | |
| 53 | Llama 3.1 8B Instruct | not supported · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); the model has no reasoning mode | 32.8 | |
| 54 | Olmo 3.1 32B Instruct | not supported · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); the model has no reasoning mode | 32.5 | |
| 55 | Command A+ (May 2026, BF16) | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 32.2 | |
| 56 | gpt-oss-20b | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 32.2 | |
| 57 | Nemotron 3.5 Lightning 30B A3B BF16 | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 28.8 | |
| 58 | GPT-6 Astra | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 27.9 | |
| 59 | Nemotron 3 Nano 30B A3B | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 27.3 | |
| 60 | Apertus 8B Instruct | not supported · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); the model has no reasoning mode | 26.0 | |
| 61 | Gemma 4 E2B IT | thinking · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); reasoning on (the board labels the row reasoning); effort or thinking budget unpublished | 25.4 | |
| 62 | Tiny Aya Global | not supported · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); the model has no reasoning mode | 15.8 | |
| 63 | Llama 3.2 3B Instruct | not supported · SEA-HELM leaderboard: zero-shot native-language prompts in seven Southeast Asian languages, 8 runs per model with the model's default generation settings (vLLM defaults otherwise); the model has no reasoning mode | 6.3 |
Benchmarks outside its family whose model rankings track it most reliably (Spearman ρ over at least 10 shared models).