IDE-Bench Pass@5
Code · 2026-01-28
Share of IDE-Bench's 80 software-engineering tasks that a model solves in at least one of five independent attempts, from the same runs as the IDE-Bench pass@1 score. Each attempt is a fresh Docker run of the IDE agent harness, graded by the task's hidden tests.
Top models (higher is better)
| Model | Score |
|---|---|
| GPT-5.2 | 95.0 |
| Sonnet 4.5 | 88.8 |
| Haiku 4.5 | 87.5 |
| Opus 4.5 | 86.3 |
| GPT-5.1-Codex-Max | 85.0 |
| Gemini 3 Pro Preview | 80.0 |
| Qwen3 Max (rolling alias) | 76.3 |
| Qwen3-Coder-480B-A35B-Instruct | 75.0 |
| DeepSeek-V3.2 | 71.3 |
| Grok 4.1 Fast | 67.5 |
| DeepSeek-R1-0528 | 46.3 |
| Grok Code Fast 1 | 32.5 |
| Llama 4 Maverick Instruct | 8.8 |
| C4AI Command R+ 08-2024 | 7.5 |
| Llama 4 Scout Instruct | 6.3 |