Benchmarks
347 sources · 1,258 models × 3,038 benchmarks · 158,912 scoresInferenceBench gives a coding agent a base model, a GPU environment and a two-hour budget to build an OpenAI-compatible inference server, and scores the final submitted server's speedup over a vanilla PyTorch baseline on four scenarios: prefill latency (time to first token), decode latency (time per output token), throughput under burst, Poisson and constant-rate traffic, and a balanced all-in-one. A server that fails the correctness checks or the integrity audit for reward hacking, is unreachable, or regresses scores at the baseline (1.0). Each scenario is run three times, and the score is the geometric mean of the four scenario speedups. These runs used the original v1.0 prompt, most of them under Claude Code v2.1.114; both the prompt and the scaffold version move scores, so this cohort is kept apart from the v1.0.1 leaderboard as a diagnostic.
No signal or difficulty estimate. Held out of the fit by design.
| # | Model | Setting | Score | Source |
|---|---|---|---|---|
| 1 | Opus 4.7 | not reported · Claude Code | 8.5 | |
| 2 | Opus 4.8 | not reported · Claude Code | 7.6 | |
| 3 | GPT-5.4 | high · Codex | 6.2 | |
| 4 | Sonnet 4.6 | not reported · Claude Code | 5.6 | |
| 5 | GPT-5.3-Codex | high · Codex | 5.5 | |
| 6 | GPT-5.5 | x-high · Codex | 5.5 | |
| 7 | Gemini 3.1 Pro Preview | not reported · OpenCode | 4.9 | |
| 8 | Kimi K2.6 | not reported · OpenCode | 4.5 | |
| 9 | Opus 4.6 | not reported · Claude Code | 4.4 | |
| 10 | GPT-5.2 | not reported · Codex CLI | 4.3 | |
| 11 | Gemini 3.5 Flash | not reported · OpenCode | 4.2 | |
| 12 | Opus 4.5 | not reported · Claude Code | 3.8 | |
| 13 | GPT-5.1-Codex-Max | not reported · Codex | 3.6 | |
| 14 | GLM-5 | not reported · OpenCode | 3.2 | |
| 15 | Sonnet 4.5 | not reported · Claude Code | 3.2 | |
| 16 | Claude Fable 5 | unknown · Claude Code | 3.2 | |
| 17 | Haiku 4.5 | not reported · Claude Code | 2.8 | |
| 18 | GPT-5.2-Codex | not reported · Codex | 2.0 |
Benchmarks outside its family whose model rankings track it most reliably (Spearman ρ over at least 10 shared models).