Benchmarks
347 sources · 1,258 models × 3,038 benchmarks · 158,912 scoresProduct Compass's Bug Hunt Bench plants 105 real bugs, mostly regressions reverted from the repositories' own fix history, in two production codebases: a roughly 28,000-line TypeScript VS Code extension (45 bugs) and a roughly 60,000-line React and Supabase learning platform (60). The test suites stay green, so nothing points at the bugs. Each model gets one identical prompt per repository in its own agentic CLI, and an independent judge, never told which model produced the diff, grades the changes against a withheld answer key. The score is the share of the 105 planted bugs fixed; partial fixes, bugs named in the report but not fixed, and fixes to unplanted defects are not counted. Many board rows average two to seven runs.
183rd strongest of 798 fitted benchmarks
Signal-to-noise +10.23
91st hardest of 798 fitted benchmarks
Capability needed 171.5
| # | Model | Setting | Score | Source |
|---|---|---|---|---|
| 1 | Claude Fable 5.2 | max · Claude Code | 55.2 | |
| 2 | Gemini 4 Carbon | high · Antigravity CLI | 54.6 | |
| 3 | Claude Sonnet 5.5 | max · Claude Code | 48.9 | |
| 4 | GPT-6 Astra | max · Codex | 42.9 | |
| 5 | GPT-6.1 Sol | max · Codex | 42.2 | |
| 6 | GPT-5.6 Sol | max · Codex | 41.4 | |
| 7 | Claude Fable 5.1 | max · Claude Code | 41.0 | |
| 8 | DeepSeek V4.1 Flash 1017 | max · DeepSeek Harness | 40.7 | |
| 9 | Muse Wonder Full | max · Muse Code | 40.0 | |
| 10 | Claude Opus 5.5 | max · Claude Code | 39.7 | |
| 11 | Gemini 4 Argon | high · Antigravity CLI | 38.7 | |
| 12 | GPT-6.1 Terra | max · Codex | 38.7 | |
| 13 | Kimi K3.1 Pro | max · Claude Code | 34.3 | |
| 14 | Gemini 4 Hydrogen | x-high · Antigravity CLI | 31.1 | |
| 15 | Muse Spark 1.3 | max · Muse Code | 30.7 | |
| 16 | GPT-5.6 Terra | max · Codex | 30.5 | |
| 17 | Muse Wonder | max · Muse Code | 30.2 | |
| 18 | GPT-5.6 Luna | max · Codex | 29.8 | |
| 19 | GPT-6 Sol | max · Codex | 27.9 | |
| 20 | Claude Fable 5 | max · Claude Code | 27.6 | |
| 21 | Grok 4.7 | x-high · Grok Build | 27.6 | |
| 22 | Grok 4.6 | x-high · Grok Build | 27.3 | |
| 23 | Claude Opus 5 | max · Claude Code | 25.7 | |
| 24 | Qwen3.8 Flash | max · Claude Code | 24.8 | |
| 25 | MiMo-V2.6-Flash | — · Claude Code | 22.2 | |
| 26 | MiMo-V2.6-Pro | not reported · Claude Code | 21.6 | |
| 27 | Gemini 3.7 Flash | high · Gemini CLI | 21.0 | |
| 28 | DeepSeek V4.1 Flash | max · Claude Code | 20.6 | |
| 29 | Claude Haiku 5.5 | max · Claude Code | 20.5 | |
| 30 | Kimi K3 | not reported · Claude Code | 20.0 | |
| 31 | GLM-5.3 | max · Claude Code | 18.1 | |
| 32 | GPT-6 Luna | max · Codex | 17.5 | |
| 33 | Gemini 3.8 Flash | high · Antigravity CLI | 17.1 | |
| 34 | Hy4 Preview | — · Claude Code | 17.1 | |
| 35 | GLM-5.3 Flash | max · Claude Code | 16.8 | |
| 36 | Grok 4.5 | high · Grok Build | 16.2 | |
| 37 | Muse Spark 1.2 | x-high · Claude Code | 16.2 | |
| 38 | Opus 4.8 | max · Claude Code | 14.3 | |
| 39 | Mistral Large 4 Preview | high · Claude Code | 14.3 | |
| 40 | Qwen3.8 27B | x-high · Claude Code | 14.3 | |
| 41 | DeepSeek-V4-Flash-0731 | not reported · Claude Code | 13.3 | |
| 42 | Sonnet 5 | max · Claude Code | 8.6 | |
| 43 | Gemma 4 31B IT | — · Claude Code | 3.8 | |
| 44 | Haiku 4.5 | not reported · Claude Code | 1.4 | |
| 45 | gpt-oss-120b | — · Claude Code | 0.0 | |
| 46 | gpt-oss-20b | — · Claude Code | 0.0 |
Benchmarks outside its family whose model rankings track it most reliably (Spearman ρ over at least 10 shared models).