Atlas

Benchmarks

Ramp SWE-Bench v2 (Bash harness)

Share of Ramp SWE-Bench v2's 500 tasks resolved in the Bash view (v2.0), where every model runs in mini-swe-agent, a minimal harness whose only tool is bash. Ramp SWE-Bench v2 (Ramp Labs) is 500 tasks synthesized from Ramp engineers' merged pull requests on Ramp's private production code, kept only where a ladder of cheap, middle-tier and frontier models failed for meaningful reasons, and graded by a panel of LLM judges that compare each candidate patch with the merged one. Every model runs each task once (pass@1) at high reasoning effort with web tools disabled; runs over 3.5 hours and persistent harness errors count as failures. A new task set and grader, not comparable with v1.

Models
22
Top score
75.0
Leader
GPT-6.2 Astra
Updated
—
Resolution
—
Saturation
—
Signal
66thpercentile
141 benchmarks from 0.8 to 2.9196 benchmarks from 2.9 to 5.0131 benchmarks from 5.0 to 7.1105 benchmarks from 7.1 to 9.282 benchmarks from 9.2 to 11.365 benchmarks from 11.3 to 13.327 benchmarks from 13.3 to 15.421 benchmarks from 15.4 to 17.58 benchmarks from 17.5 to 19.68 benchmarks from 19.6 to 21.76 benchmarks from 21.7 to 23.84 benchmarks from 23.8 to 25.95 benchmarks from 25.9 to 28.03 benchmarks from 28.0 to 30.12 benchmarks from 30.1 to 32.14 benchmarks from 32.1 to 34.21 benchmark from 34.2 to 36.33 benchmarks from 36.3 to 38.41 benchmark from 38.4 to 40.52 benchmarks from 40.5 to 42.61 benchmark from 42.6 to 44.71 benchmark from 44.7 to 46.81 benchmark from 48.9 to 50.9

280th strongest of 818

Signal-to-noise +8.27 for the Ramp SWE-Bench v2 family

Difficulty
Not estimated

Another benchmark represents this family in the fit.

Organizations
22 dated models · 4 frontier modelsClosed weightsOpen weights
Data: Ramp Labsatlas.village.dev

Ramp SWE-Bench v2 (Bash harness) leaderboard

#ModelSettingScoreSource
1GPT-6.2 Astrahigh · Mini-SWE-Agent
75.0
2Gemini 4 Carbonhigh · Mini-SWE-Agent
74.0
3Claude Fable 5.2high · Mini-SWE-Agent
73.0
4GPT-6.2 Solhigh · Mini-SWE-Agent
72.4
5Claude Opus 5.5high · Mini-SWE-Agent
71.2
6Gemini 4 Hydrogenhigh · Mini-SWE-Agent
70.8
7Claude Fable 5.1high · Mini-SWE-Agent
63.8
8GPT-6.1 Terrahigh · Mini-SWE-Agent
60.6
9Muse Wonderhigh · Mini-SWE-Agent
57.8
10Claude Sonnet 5.5high · Mini-SWE-Agent
57.6
11Grok 4.7high · Mini-SWE-Agent
55.4
12Gemini 4 Argonhigh · Mini-SWE-Agent
54.8
13GPT-6 Astrahigh · Mini-SWE-Agent
54.6
14Muse Emberhigh · Mini-SWE-Agent
54.4
15DeepSeek V4.1 Flash 1017high · Mini-SWE-Agent
53.4
16Claude Haiku 5.5high · Mini-SWE-Agent
52.8
17GPT-6.1 Solhigh · Mini-SWE-Agent
52.0
18Gemini 3.8 Flashhigh · Mini-SWE-Agent
47.0
19Kimi K3.1 Prohigh · Mini-SWE-Agent
45.4
20GPT-6 Lunahigh · Mini-SWE-Agent
42.6
21Beam 1.0 501B A23Bhigh · Mini-SWE-Agent
34.0
22Nemotron 3.5 Super VL 120B A12B (precision unspecified)not reported · Mini-SWE-Agent; source effort high, native translation unspecified
12.0

Related benchmarks

Loading Atlas data…