Benchmarks
347 sources · 1,258 models × 3,038 benchmarks · 158,912 scoresUser Correction on SWE-Together: for each run, the number of the simulated user's messages tagged as corrections plus 0.2 for each implicit nudge, averaged over the two runs per task and then over the 109 tasks; lower means the agent needed less steering. A Gemini 3.1 Pro tagger labels the messages of a Gemini 3.1 Pro user simulator on this board, so the measure depends on those seats as well as the agent. Diagnostic; the primary `swe-together` is pass@1.
No signal or difficulty estimate. Held out of the fit by design.
| # | Model | Setting | Score | Source |
|---|---|---|---|---|
| 1 | Grok 4.6 | high · OpenCode | 1.4 | |
| 2 | Claude Opus 5 | high · OpenCode | 1.4 | |
| 3 | Claude Fable 5.1 | high · OpenCode | 1.5 | |
| 4 | Grok 4.7 | high · OpenCode | 1.5 | |
| 5 | Claude Opus 5.5 | high · OpenCode | 1.5 | |
| 6 | Claude Fable 5 | high · OpenCode | 1.5 | |
| 7 | Gemini 3.8 Flash | high · OpenCode | 1.6 | |
| 8 | GPT-6 Astra | high · OpenCode | 1.6 | |
| 9 | GPT-6.1 Sol | high · OpenCode | 1.6 | |
| 10 | Opus 4.8 | high · OpenCode | 1.6 | |
| 11 | GPT-5.6 Sol | high · OpenCode | 1.6 | |
| 12 | Claude Sonnet 5.5 | high · OpenCode | 1.7 | |
| 13 | Opus 4.7 | high · OpenCode | 1.7 | |
| 14 | Opus 4.6 | high · OpenCode | 1.7 | |
| 15 | Kimi K3 | high · OpenCode | 1.7 | |
| 16 | GPT-6 Luna | high · OpenCode | 1.8 | |
| 17 | GPT-6 Sol | high · OpenCode | 1.9 | |
| 18 | Muse Spark 1.3 | high · OpenCode | 1.9 |
Benchmarks outside its family whose model rankings track it most reliably (Spearman ρ over at least 10 shared models).