IDE-Bench
Code · 2026-01-28
AfterQuery's IDE-Bench evaluates models as IDE agents on 80 software-engineering tasks (feature work, bug fixes, refactoring and performance optimization) in eight never-published repositories spanning C/C++, Java, Python and TypeScript/MERN stacks. Each attempt runs in a fresh Docker container where the agent works through 17 IDE-style tools (file navigation and codebase search, structured editing, a terminal, and API and database calls for full-stack tasks) for up to 100 iterations, and it succeeds when the task's hidden tests pass. The score is pass@1, the share of tasks solved on the first of five independent attempts.
Top models (higher is better)
| Model | Score |
|---|---|
| Sonnet 4.5 | 87.5 |
| GPT-5.2 | 85.0 |
| Opus 4.5 | 83.8 |
| Haiku 4.5 | 78.8 |
| GPT-5.1-Codex-Max | 73.8 |
| Qwen3 Max (rolling alias) | 65.0 |
| Qwen3-Coder-480B-A35B-Instruct | 57.5 |
| Gemini 3 Pro Preview | 55.0 |
| Grok 4.1 Fast | 35.0 |
| DeepSeek-V3.2 | 31.3 |
| DeepSeek-R1-0528 | 20.0 |
| Grok Code Fast 1 | 11.3 |
| Llama 4 Maverick Instruct | 2.5 |
| Llama 4 Scout Instruct | 2.5 |
| C4AI Command R+ 08-2024 | 0.0 |