CompileBench
Code · 2025-09-17
Quesma's CompileBench gives an agent the unmodified source of an open-source project (curl, jq, GNU coreutils, cowsay) in an interactive Docker Linux terminal and asks for a working binary: 15 tasks ranging from plain builds to static and musl linking, cross-compiling for ARM64 and Windows, and reviving 2003-era coreutils 5.0. Task-specific checks verify that the produced binary is genuine and works. Models act through function calls in Quesma's own minimal harness via OpenRouter, three attempts per task, and the score is the share of the 45 attempts that succeed (pass@1).
Top models (higher is better)
| Model | Score |
|---|---|
| Sonnet 4 | 91.1 |
| GPT-5-Codex | 91.1 |
| Sonnet 4.5 | 86.7 |
| GPT-5 | 86.7 |
| Opus 4.1 | 80.0 |
| GPT-5 Mini | 75.6 |
| Haiku 4.5 | 73.3 |
| Grok 4 | 71.1 |
| DeepSeek-V3.1 | 64.4 |
| DeepSeek-V3.1-Terminus | 64.4 |
| Grok Code Fast 1 | 64.4 |
| Kimi K2 Instruct 0905 | 57.8 |
| Grok 4 Fast | 55.6 |
| Gemini 2.5 Pro | 53.3 |
| GPT-4.1 | 53.3 |
| Qwen3 Max (rolling alias) | 53.3 |
| GLM-4.5 | 48.9 |
| GPT-4.1 Mini | 46.7 |
| Gemini 2.5 Flash | 44.4 |
| gpt-oss-120b | 42.2 |