CompileBench (pass@3)
Code · 2025-09-17
Share of CompileBench's 15 build tasks (open-source projects compiled from unmodified source in a Docker terminal, verified by task-specific binary checks) that a model completes in at least one of its three attempts. It comes from the same attempts as the pass@1 headline and is a component of its factor group.
Top models (higher is better)
| Model | Score |
|---|---|
| Opus 4.1 | 100.0 |
| Sonnet 4 | 93.3 |
| GPT-5-Codex | 93.3 |
| GPT-5 | 93.3 |
| Haiku 4.5 | 86.7 |
| Sonnet 4.5 | 86.7 |
| Grok 4 | 86.7 |
| DeepSeek-V3.1 | 80.0 |
| GPT-5 Mini | 80.0 |
| Kimi K2 Instruct 0905 | 80.0 |
| DeepSeek-V3.1-Terminus | 73.3 |
| GLM-4.5 | 73.3 |
| Grok 4 Fast | 73.3 |
| Grok Code Fast 1 | 73.3 |
| GPT-4.1 Mini | 66.7 |
| Qwen3 Max (rolling alias) | 66.7 |
| Gemini 2.5 Flash | 60.0 |
| Gemini 2.5 Pro | 60.0 |
| GPT-4.1 | 60.0 |
| gpt-oss-120b | 60.0 |