MacroscopeBench
Code · 2026-09-22
Macroscope's code-review benchmark. Each task is one commit from a public open-source repository: the model reviews the commit's diff against its parent in a checkout of the repository, without being told whether a bug exists, and two LLM judges match its findings to known bugs (defects maintainers later fixed) and grade every finding's validity. The dataset is proprietary and includes control commits with no known bug; the public leaderboard does not say which commits it covers. The score is the harmonic mean (F1) of known-bug recall and overall precision from three runs per model, in percent.
Top models (higher is better)
| Model | Score |
|---|---|
| Claude Opus 5.5 | 83.9 |
| GPT-6.1 Terra | 82.3 |
| GPT-6 Sol | 80.3 |
| GPT-6.1 Sol | 78.5 |
| GPT-6 Astra | 78.0 |
| GPT-5.6 Sol | 77.8 |
| GLM-5.3 | 77.4 |
| Grok 4.6 | 75.9 |
| Kimi K3 | 72.9 |
| GPT-6 Luna | 72.5 |
| DeepSeek V4.1 Flash | 72.0 |
| Gemini 3.8 Flash | 70.7 |
| Claude Opus 5 | 69.9 |
| Muse Wonder | 64.2 |