Benchmarks
347 sources · 1,258 models × 3,038 benchmarks · 158,912 scoresThe Last Translation Benchmark (Zouhar et al., ETH Zurich and a crowd of co-authors) collects human-authored, peer-reviewed inputs that break leading machine translation models, each with a reference translation and handwritten verification rules naming its failure cases; an example was accepted only if at most two of the up to ten models shown to contributors passed its rules. This board scores blind translations of the 911 text-only LTBv1-eval examples, which span 39 source and 14 target languages. A Gemini 3.8 Flash verifier checks each rule, and the score is the share of examples whose translation passes every rule; an untranslated example fails. The ten models shown to contributors were scored on the very translations the examples were selected against, so their scores are biased down.
471st strongest of 798 fitted benchmarks
Signal-to-noise +4.88
50th hardest of 798 fitted benchmarks
Capability needed ≳ 178.3
| # | Model | Setting | Score | Source |
|---|---|---|---|---|
| 1 | GPT-6 Astra Pro | not reported | 45.3 | |
| 2 | GPT-6 Astra | not reported | 44.0 | |
| 3 | Gemini 4 Argon | not reported | 40.9 | |
| 4 | Gemini 3.8 Flash | not reported | 39.8 | |
| 5 | Claude Opus 5.5 | medium | 35.2 | |
| 6 | Claude Fable 5.1 | not reported | 32.9 | |
| 7 | Muse Spark 1.3 | not reported | 18.4 | |
| 8 | GPT-5.6 Luna | not reported | 15.4 | |
| 9 | Qwen3.8 Max (0902) | not reported | 12.5 | |
| 10 | GPT-5.6 Terra | not reported | 12.2 | |
| 11 | Kimi K3 | not reported | 11.7 | |
| 12 | Qwen3.7-Plus (source unspecified) | — | 10.2 | |
| 13 | GLM-5.3 | not reported | 8.6 | |
| 14 | MiniMax M3 | — | 6.3 | |
| 15 | Qwen3.7 Flash | not reported | 5.8 | |
| 16 | Nemotron 3 Ultra 550B A55B | not reported | 4.8 | |
| 17 | Gemini 2.5 Flash | — | 4.6 | |
| 18 | gpt-oss-120b | not reported | 2.3 | |
| 19 | Command A+ (May 2026, BF16) | — | 1.4 | |
| 20 | gpt-oss-20b | not reported | 1.3 | |
| 21 | Mistral Small 4 | — | 1.3 | |
| 22 | Apertus 70B Instruct | not supported | 1.2 | |
| 23 | Apertus 8B Instruct | not supported | 0.8 | |
| 24 | Command A | not supported | 0.7 | |
| 25 | Tiny Aya Global | not supported | 0.7 |
Benchmarks outside its family whose model rankings track it most reliably (Spearman ρ over at least 10 shared models).