Atlas

Benchmarks

Bug Hunt Bench

Product Compass's Bug Hunt Bench plants 105 real bugs, mostly regressions reverted from the repositories' own fix history, in two production codebases: a roughly 28,000-line TypeScript VS Code extension (45 bugs) and a roughly 60,000-line React and Supabase learning platform (60). The test suites stay green, so nothing points at the bugs. Each model gets one identical prompt per repository in its own agentic CLI, and an independent judge, never told which model produced the diff, grades the changes against a withheld answer key. The score is the share of the 105 planted bugs fixed; partial fixes, bugs named in the report but not fixed, and fixes to unplanted defects are not counted. Many board rows average two to seven runs.

Models
46
Top score
55.2
Leader
Claude Fable 5.2
Updated
Nov 13, 2026
Resolution
~2 mo
Sharp
Saturation
~Jan 2028
Projected
Signal
77thpercentile
195 benchmarks from 0.8 to 3.5224 benchmarks from 3.5 to 6.1133 benchmarks from 6.1 to 8.898 benchmarks from 8.8 to 11.566 benchmarks from 11.5 to 14.126 benchmarks from 14.1 to 16.815 benchmarks from 16.8 to 19.59 benchmarks from 19.5 to 22.18 benchmarks from 22.1 to 24.84 benchmarks from 24.8 to 27.55 benchmarks from 27.5 to 30.12 benchmarks from 30.1 to 32.84 benchmarks from 32.8 to 35.45 benchmarks from 35.4 to 38.11 benchmark from 38.1 to 40.82 benchmarks from 43.4 to 46.11 benchmark from 62.1 to 64.8

183rd strongest of 798

Signal-to-noise +10.23

Difficulty
88thpercentile
1 benchmark from 5.5 to 14.41 benchmark from 14.4 to 23.22 benchmarks from 32.1 to 41.01 benchmark from 41.0 to 49.82 benchmarks from 49.8 to 58.79 benchmarks from 58.7 to 67.67 benchmarks from 67.6 to 76.418 benchmarks from 76.4 to 85.324 benchmarks from 85.3 to 94.241 benchmarks from 94.2 to 103.031 benchmarks from 103.0 to 111.939 benchmarks from 111.9 to 120.850 benchmarks from 120.8 to 129.676 benchmarks from 129.6 to 138.560 benchmarks from 138.5 to 147.4118 benchmarks from 147.4 to 156.2124 benchmarks from 156.2 to 165.1120 benchmarks from 165.1 to 174.039 benchmarks from 174.0 to 182.820 benchmarks from 182.8 to 191.710 benchmarks from 191.7 to 200.61 benchmark from 200.6 to 209.44 benchmarks from 209.4 to 218.3

91st hardest of 798

Capability needed 171.5

Organizations
46 dated models · 10 frontier modelsClosed weightsOpen weights
Data: Product Compassatlas.village.dev

Bug Hunt Bench leaderboard

#ModelSettingScoreSource
1Claude Fable 5.2max · Claude Code
55.2
2Gemini 4 Carbonhigh · Antigravity CLI
54.6
3Claude Sonnet 5.5max · Claude Code
48.9
4GPT-6 Astramax · Codex
42.9
5GPT-6.1 Solmax · Codex
42.2
6GPT-5.6 Solmax · Codex
41.4
7Claude Fable 5.1max · Claude Code
41.0
8DeepSeek V4.1 Flash 1017max · DeepSeek Harness
40.7
9Muse Wonder Fullmax · Muse Code
40.0
10Claude Opus 5.5max · Claude Code
39.7
11Gemini 4 Argonhigh · Antigravity CLI
38.7
12GPT-6.1 Terramax · Codex
38.7
13Kimi K3.1 Promax · Claude Code
34.3
14Gemini 4 Hydrogenx-high · Antigravity CLI
31.1
15Muse Spark 1.3max · Muse Code
30.7
16GPT-5.6 Terramax · Codex
30.5
17Muse Wondermax · Muse Code
30.2
18GPT-5.6 Lunamax · Codex
29.8
19GPT-6 Solmax · Codex
27.9
20Claude Fable 5max · Claude Code
27.6
21Grok 4.7x-high · Grok Build
27.6
22Grok 4.6x-high · Grok Build
27.3
23Claude Opus 5max · Claude Code
25.7
24Qwen3.8 Flashmax · Claude Code
24.8
25MiMo-V2.6-Flash— · Claude Code
22.2
26MiMo-V2.6-Pronot reported · Claude Code
21.6
27Gemini 3.7 Flashhigh · Gemini CLI
21.0
28DeepSeek V4.1 Flashmax · Claude Code
20.6
29Claude Haiku 5.5max · Claude Code
20.5
30Kimi K3not reported · Claude Code
20.0
31GLM-5.3max · Claude Code
18.1
32GPT-6 Lunamax · Codex
17.5
33Gemini 3.8 Flashhigh · Antigravity CLI
17.1
34Hy4 Preview— · Claude Code
17.1
35GLM-5.3 Flashmax · Claude Code
16.8
36Grok 4.5high · Grok Build
16.2
37Muse Spark 1.2x-high · Claude Code
16.2
38Opus 4.8max · Claude Code
14.3
39Mistral Large 4 Previewhigh · Claude Code
14.3
40Qwen3.8 27Bx-high · Claude Code
14.3
41DeepSeek-V4-Flash-0731not reported · Claude Code
13.3
42Sonnet 5max · Claude Code
8.6
43Gemma 4 31B IT— · Claude Code
3.8
44Haiku 4.5not reported · Claude Code
1.4
45gpt-oss-120b— · Claude Code
0.0
46gpt-oss-20b— · Claude Code
0.0

Related benchmarks

Loading Atlas data…