Benchmarks
357 sources · 1,263 models × 3,079 benchmarks · 162,127 scoresShare of Ramp SWE-Bench v2's 500 tasks resolved in the Bash view (v2.0), where every model runs in mini-swe-agent, a minimal harness whose only tool is bash. Ramp SWE-Bench v2 (Ramp Labs) is 500 tasks synthesized from Ramp engineers' merged pull requests on Ramp's private production code, kept only where a ladder of cheap, middle-tier and frontier models failed for meaningful reasons, and graded by a panel of LLM judges that compare each candidate patch with the merged one. Every model runs each task once (pass@1) at high reasoning effort with web tools disabled; runs over 3.5 hours and persistent harness errors count as failures. A new task set and grader, not comparable with v1.
280th strongest of 818 fitted benchmarks
Signal-to-noise +8.27 for the Ramp SWE-Bench v2 family
Another benchmark represents this family in the fit.
| # | Model | Setting | Score | Source |
|---|---|---|---|---|
| 1 | GPT-6.2 Astra | high · Mini-SWE-Agent | 75.0 | |
| 2 | Gemini 4 Carbon | high · Mini-SWE-Agent | 74.0 | |
| 3 | Claude Fable 5.2 | high · Mini-SWE-Agent | 73.0 | |
| 4 | GPT-6.2 Sol | high · Mini-SWE-Agent | 72.4 | |
| 5 | Claude Opus 5.5 | high · Mini-SWE-Agent | 71.2 | |
| 6 | Gemini 4 Hydrogen | high · Mini-SWE-Agent | 70.8 | |
| 7 | Claude Fable 5.1 | high · Mini-SWE-Agent | 63.8 | |
| 8 | GPT-6.1 Terra | high · Mini-SWE-Agent | 60.6 | |
| 9 | Muse Wonder | high · Mini-SWE-Agent | 57.8 | |
| 10 | Claude Sonnet 5.5 | high · Mini-SWE-Agent | 57.6 | |
| 11 | Grok 4.7 | high · Mini-SWE-Agent | 55.4 | |
| 12 | Gemini 4 Argon | high · Mini-SWE-Agent | 54.8 | |
| 13 | GPT-6 Astra | high · Mini-SWE-Agent | 54.6 | |
| 14 | Muse Ember | high · Mini-SWE-Agent | 54.4 | |
| 15 | DeepSeek V4.1 Flash 1017 | high · Mini-SWE-Agent | 53.4 | |
| 16 | Claude Haiku 5.5 | high · Mini-SWE-Agent | 52.8 | |
| 17 | GPT-6.1 Sol | high · Mini-SWE-Agent | 52.0 | |
| 18 | Gemini 3.8 Flash | high · Mini-SWE-Agent | 47.0 | |
| 19 | Kimi K3.1 Pro | high · Mini-SWE-Agent | 45.4 | |
| 20 | GPT-6 Luna | high · Mini-SWE-Agent | 42.6 | |
| 21 | Beam 1.0 501B A23B | high · Mini-SWE-Agent | 34.0 | |
| 22 | Nemotron 3.5 Super VL 120B A12B (precision unspecified) | not reported · Mini-SWE-Agent; source effort high, native translation unspecified | 12.0 |
Benchmarks outside its family whose model rankings track it most reliably (Spearman ρ over at least 10 shared models).