Benchmarks
326 sources · 1,256 models × 2,956 benchmarks · 154,484 scoresDeepSWE v1.1 (Harness Leaderboard, avg@3)
The Harness Leaderboard's run of DeepSWE v1.1 (Datacurve's commit 0b9fabbb) on 111 of its 113 tasks, comparing coding-agent harnesses on fixed open models in Harbor; helm-unified-manifest-stream, whose reference solution fails on Hugging Face's sandboxes, and vulture-persistent-analysis-cache, whose grader fails correct solutions, are left out. Each task scores the mean of three attempts and the rate is the mean over tasks. Infrastructure failures are retried and excluded, trials that looked up a task's answer key count as failures, and trials ran with internet access until October 23, 2026 and offline after. Kept as a component beside Datacurve's own pooled pass@1.
170th strongest of 779 fitted benchmarks
Signal-to-noise +10.55 for the DeepSWE v1.1 family
Another benchmark represents this family in the fit.
| # | Model | Setting | Score | Source |
|---|---|---|---|---|
| 1 | DeepSeek V4.1 Flash | high · Harness Leaderboard; Hermes Agent v0.22.1 in Harbor | 75.1 | |
| 2 | GLM-5.3 Flash | high · Harness Leaderboard; Kimi Code 2.2.0 in Harbor; told every model holds 1,000,000 tokens | 63.7 | |
| 3 | Flint | always on · Harness Leaderboard; Kimi Code 2.2.0 in Harbor; told every model holds 1,000,000 tokens | 11.1 |
