SWE-Marathon v1.0
Code · 2026-06-05
Archived SWE-Marathon v1.0 evaluates agents on 20 executable, ultra-long-horizon tasks spanning software engineering and adjacent technical domains, with five nominal trials per task. Scores average binary success across repeated trials. Distinct from the current v1.1 task environments and eight-trial protocol.
Top models (higher is better)
| Model | Score |
|---|---|
| Grok 4.5 | 29.0 |
| Opus 4.8 | 26.0 |
| Claude Fable 5 | 24.0 |
| Opus 4.7 | 16.0 |
| GPT-5.5 | 14.0 |
| GLM-5.2 | 13.0 |
| Gemini 3.5 Flash | 7.0 |
| DeepSeek-V4-Pro | 4.0 |
| Gemini 3.1 Pro Preview | 4.0 |
| GLM-5.1 | 1.0 |
| Kimi K2.6 | 0.0 |
| MiniMax M2.7 | 0.0 |