Atlas

Benchmarks

InferenceBench (prompt v1.0)

InferenceBench gives a coding agent a base model, a GPU environment and a two-hour budget to build an OpenAI-compatible inference server, and scores the final submitted server's speedup over a vanilla PyTorch baseline on four scenarios: prefill latency (time to first token), decode latency (time per output token), throughput under burst, Poisson and constant-rate traffic, and a balanced all-in-one. A server that fails the correctness checks or the integrity audit for reward hacking, is unreachable, or regresses scores at the baseline (1.0). Each scenario is run three times, and the score is the geometric mean of the four scenario speedups. These runs used the original v1.0 prompt, most of them under Claude Code v2.1.114; both the prompt and the scaffold version move scores, so this cohort is kept apart from the v1.0.1 leaderboard as a diagnostic.

Models
18
Top score
8.5
Leader
Opus 4.7
Updated
—
Resolution
—
Saturation
—

No signal or difficulty estimate. Held out of the fit by design.

Organizations
18 dated models · 8 frontier modelsClosed weightsOpen weights
Data: InferenceBenchatlas.village.dev

InferenceBench (prompt v1.0) leaderboard

Related benchmarks

Most correlated

Benchmarks outside its family whose model rankings track it most reliably (Spearman ρ over at least 10 shared models).

Loading Atlas data…