Atlas

Benchmarks

Ramp SWE-Bench v2

Share of Ramp SWE-Bench v2's 500 tasks resolved in the default Native view (v2.1), where each model runs in its lab's own agent harness, or OpenCode where that harness isn't compatible with Ramp Router. Ramp SWE-Bench v2 (Ramp Labs) is 500 tasks synthesized from Ramp engineers' merged pull requests on Ramp's private production code, kept only where a ladder of cheap, middle-tier and frontier models failed for meaningful reasons, and graded by a panel of LLM judges that compare each candidate patch with the merged one. Every model runs each task once (pass@1) at high reasoning effort with web tools disabled; runs over 3.5 hours and persistent harness errors count as failures. A new task set and grader, not comparable with v1.

Models
16
Top score
76.4
Leader
GPT-6.2 Astra
Updated
—
Resolution
~3 mo
Sharp
Saturation
~Nov 2027
Projected
Signal
66thpercentile
141 benchmarks from 0.8 to 2.9196 benchmarks from 2.9 to 5.0131 benchmarks from 5.0 to 7.1105 benchmarks from 7.1 to 9.282 benchmarks from 9.2 to 11.365 benchmarks from 11.3 to 13.327 benchmarks from 13.3 to 15.421 benchmarks from 15.4 to 17.58 benchmarks from 17.5 to 19.68 benchmarks from 19.6 to 21.76 benchmarks from 21.7 to 23.84 benchmarks from 23.8 to 25.95 benchmarks from 25.9 to 28.03 benchmarks from 28.0 to 30.12 benchmarks from 30.1 to 32.14 benchmarks from 32.1 to 34.21 benchmark from 34.2 to 36.33 benchmarks from 36.3 to 38.41 benchmark from 38.4 to 40.52 benchmarks from 40.5 to 42.61 benchmark from 42.6 to 44.71 benchmark from 44.7 to 46.81 benchmark from 48.9 to 50.9

280th strongest of 818

Signal-to-noise +8.27

Difficulty
56thpercentile
1 benchmark from 5.8 to 14.71 benchmark from 23.5 to 32.42 benchmarks from 32.4 to 41.21 benchmark from 41.2 to 50.12 benchmarks from 50.1 to 58.99 benchmarks from 58.9 to 67.88 benchmarks from 67.8 to 76.718 benchmarks from 76.7 to 85.523 benchmarks from 85.5 to 94.441 benchmarks from 94.4 to 103.232 benchmarks from 103.2 to 112.141 benchmarks from 112.1 to 121.049 benchmarks from 121.0 to 129.875 benchmarks from 129.8 to 138.760 benchmarks from 138.7 to 147.5122 benchmarks from 147.5 to 156.4135 benchmarks from 156.4 to 165.3124 benchmarks from 165.3 to 174.139 benchmarks from 174.1 to 183.021 benchmarks from 183.0 to 191.88 benchmarks from 191.8 to 200.72 benchmarks from 200.7 to 209.54 benchmarks from 209.5 to 218.4

363rd hardest of 818

Capability needed ≲ 154.6

Organizations
16 dated models · 6 frontier modelsClosed weightsOpen weights
Data: Ramp Labsatlas.village.dev

Ramp SWE-Bench v2 leaderboard

Related benchmarks

Loading Atlas data…