Benchmarks
357 sources · 1,263 models × 3,079 benchmarks · 162,127 scoresShare of Ramp SWE-Bench v2's 500 tasks resolved in the default Native view (v2.1), where each model runs in its lab's own agent harness, or OpenCode where that harness isn't compatible with Ramp Router. Ramp SWE-Bench v2 (Ramp Labs) is 500 tasks synthesized from Ramp engineers' merged pull requests on Ramp's private production code, kept only where a ladder of cheap, middle-tier and frontier models failed for meaningful reasons, and graded by a panel of LLM judges that compare each candidate patch with the merged one. Every model runs each task once (pass@1) at high reasoning effort with web tools disabled; runs over 3.5 hours and persistent harness errors count as failures. A new task set and grader, not comparable with v1.
280th strongest of 818 fitted benchmarks
Signal-to-noise +8.27
363rd hardest of 818 fitted benchmarks
Capability needed ≲ 154.6
| # | Model | Setting | Score | Source |
|---|---|---|---|---|
| 1 | GPT-6.2 Astra | high · Codex | 76.4 | |
| 2 | Gemini 4 Carbon | high · OpenCode | 75.4 | |
| 3 | Claude Fable 5.2 | high · Claude Code | 73.4 | |
| 4 | GPT-6.2 Sol | high · Codex | 73.4 | |
| 5 | Gemini 4 Hydrogen | high · OpenCode | 71.6 | |
| 6 | GPT-6.1 Terra | high · Codex | 70.4 | |
| 7 | Claude Opus 5.5 | high · Claude Code | 67.8 | |
| 8 | Grok 4.7 | high · Grok Build | 64.4 | |
| 9 | GPT-6 Astra | high · Codex | 63.4 | |
| 10 | Claude Sonnet 5.5 | high · Claude Code | 63.2 | |
| 11 | DeepSeek V4.1 Flash 1017 | high · DeepSeek Harness | 60.8 | |
| 12 | GPT-6.1 Sol | high · Codex | 59.6 | |
| 13 | Muse Wonder | high · Muse Code | 58.6 | |
| 14 | Claude Haiku 5.5 | high · Claude Code | 57.4 | |
| 15 | GPT-6 Luna | high · Codex | 52.4 | |
| 16 | Muse Ember | high · Muse Code | 51.8 |
Benchmarks outside its family whose model rankings track it most reliably (Spearman ρ over at least 10 shared models).