Atlas

Benchmarks

Last Translation Benchmark (LTBv1-eval, blind)

The Last Translation Benchmark (Zouhar et al., ETH Zurich and a crowd of co-authors) collects human-authored, peer-reviewed inputs that break leading machine translation models, each with a reference translation and handwritten verification rules naming its failure cases; an example was accepted only if at most two of the up to ten models shown to contributors passed its rules. This board scores blind translations of the 911 text-only LTBv1-eval examples, which span 39 source and 14 target languages. A Gemini 3.8 Flash verifier checks each rule, and the score is the share of examples whose translation passes every rule; an untranslated example fails. The ten models shown to contributors were scored on the very translations the examples were selected against, so their scores are biased down.

Models
25
Top score
45.3
Leader
GPT-6 Astra Pro
Updated
—
Resolution
~4 mo
Useful
Saturation
~Sep 2028
Projected
Signal
41stpercentile
195 benchmarks from 0.8 to 3.5224 benchmarks from 3.5 to 6.1133 benchmarks from 6.1 to 8.898 benchmarks from 8.8 to 11.566 benchmarks from 11.5 to 14.126 benchmarks from 14.1 to 16.815 benchmarks from 16.8 to 19.59 benchmarks from 19.5 to 22.18 benchmarks from 22.1 to 24.84 benchmarks from 24.8 to 27.55 benchmarks from 27.5 to 30.12 benchmarks from 30.1 to 32.84 benchmarks from 32.8 to 35.45 benchmarks from 35.4 to 38.11 benchmark from 38.1 to 40.82 benchmarks from 43.4 to 46.11 benchmark from 62.1 to 64.8

471st strongest of 798

Signal-to-noise +4.88

Difficulty
94thpercentile
1 benchmark from 5.5 to 14.41 benchmark from 14.4 to 23.22 benchmarks from 32.1 to 41.01 benchmark from 41.0 to 49.82 benchmarks from 49.8 to 58.79 benchmarks from 58.7 to 67.67 benchmarks from 67.6 to 76.418 benchmarks from 76.4 to 85.324 benchmarks from 85.3 to 94.241 benchmarks from 94.2 to 103.031 benchmarks from 103.0 to 111.939 benchmarks from 111.9 to 120.850 benchmarks from 120.8 to 129.676 benchmarks from 129.6 to 138.560 benchmarks from 138.5 to 147.4118 benchmarks from 147.4 to 156.2124 benchmarks from 156.2 to 165.1120 benchmarks from 165.1 to 174.039 benchmarks from 174.0 to 182.820 benchmarks from 182.8 to 191.710 benchmarks from 191.7 to 200.61 benchmark from 200.6 to 209.44 benchmarks from 209.4 to 218.3

50th hardest of 798

Capability needed ≳ 178.3

Organizations
24 dated models · 7 frontier modelsClosed weightsOpen weights
Data: ETH Zurichatlas.village.dev

Last Translation Benchmark (LTBv1-eval, blind) leaderboard

#ModelSettingScoreSource
1GPT-6 Astra Pronot reported
45.3
2GPT-6 Astranot reported
44.0
3Gemini 4 Argonnot reported
40.9
4Gemini 3.8 Flashnot reported
39.8
5Claude Opus 5.5medium
35.2
6Claude Fable 5.1not reported
32.9
7Muse Spark 1.3not reported
18.4
8GPT-5.6 Lunanot reported
15.4
9Qwen3.8 Max (0902)not reported
12.5
10GPT-5.6 Terranot reported
12.2
11Kimi K3not reported
11.7
12Qwen3.7-Plus (source unspecified)—
10.2
13GLM-5.3not reported
8.6
14MiniMax M3—
6.3
15Qwen3.7 Flashnot reported
5.8
16Nemotron 3 Ultra 550B A55Bnot reported
4.8
17Gemini 2.5 Flash—
4.6
18gpt-oss-120bnot reported
2.3
19Command A+ (May 2026, BF16)—
1.4
20gpt-oss-20bnot reported
1.3
21Mistral Small 4—
1.3
22Apertus 70B Instructnot supported
1.2
23Apertus 8B Instructnot supported
0.8
24Command Anot supported
0.7
25Tiny Aya Globalnot supported
0.7

Related benchmarks

Most correlated

Benchmarks outside its family whose model rankings track it most reliably (Spearman ρ over at least 10 shared models).

Loading Atlas data…