Methodology
Atlas turns thousands of public benchmark results into two numbers per model: a capability score for how well it performs across benchmarks, and a relative cost for what it costs to run. This page explains what those numbers mean, how they are estimated, and how well they predict results they were not fitted on.
What the scores mean
Capability score. How well a model is likely to perform on benchmarks in general. It is a relative scale: two reference models fix its units, and every other model is placed against them. Differences between models are meaningful, but the numbers are not an absolute measure of intelligence.
Models are often run at several reasoning settings (reasoning off, or low, medium or high effort). Atlas estimates a capability for each reasoning state, a model at one setting, and a model’s headline score is its best state. Every score carries an uncertainty interval. A wide interval usually means few benchmarks, not a weak model.
Relative cost. What a reasoning state costs to run per task, as a multiple of GPT-5 (high): 0.5× is half its cost, 3× is three times. Alongside it, Atlas shows typical dollars per task on a short-answer workload (short questions) and an agentic one (long agent tasks).
Where the data comes from
Scores come from three kinds of source, kept separate: model releases (a lab’s own launch posts, papers and model cards), evaluator releases (independent organizations running their own harnesses), and benchmark releases (leaderboards kept by benchmark authors). Every score is tied to an exact model configuration, an exact benchmark variant and its source.
Before fitting, Atlas cleans the data rather than averaging it. Reports of the same result that agree within sampling noise are merged; reports that genuinely conflict are dropped and logged, not averaged. Each benchmark is read under one evaluation setup, because the same test run with tools or a different harness is a different measurement. Composite indices are left out, since their inputs are already benchmark results, and so are safety benchmarks, which measure behaviour rather than capability. The full rules are under Technical details.
How capability is estimated
Benchmarks use different scales, have different amounts of noise and separate strong models from weak ones with different sharpness, so Atlas does not average their scores. Instead each benchmark gets its own curve from capability to expected score, and the model fits all the curves and all the capabilities together. A model config is a model at one reasoning setting.
The curve. On a bounded benchmark, the expected score rises along an S-curve from the guessing floor to a perfect score. A score at chance says nothing about capability, and the floor is what lets the model say so.
predicted_score = chance_floor + (1 − chance_floor) × sigmoid( … )
│ │ │ │
│ │ │ └── S-curve that maps any number to 0..1
│ │ │ (its input is broken down next)
│ │ │
│ │ └── room above the floor that skill can earn
│ │
│ └── score from guessing: e.g. 0.25 for 4-option multiple choice,
│ 0 when none is known (one per benchmark, fixed, not learned)
│
└── the expected score; actual scores scatter around it with noiseWhat goes into the curve is how far the model config sits above or below the benchmark’s difficulty, stretched by how sharply the benchmark separates models. A sharp benchmark tells you far more per result than a flat one. Difficulty is shown in the benchmark catalog.
sigmoid( benchmark_steepness × (capability_on_benchmark − benchmark_difficulty) )
│ │ │
│ │ └── capability where a model scores
│ │ halfway up the curve; on the same
│ │ scale as capability
│ │ (one per benchmark, learned)
│ │
│ └── how good this model config is on this benchmark
│ (broken down below)
│
└── how sharply scores rise with capability; always > 0
(one per benchmark, learned)Ratings. Elo-style ratings are already on an open-ended scale, so they skip the floor and the S-curve and are read straight off the same line.
predicted_rating = benchmark_steepness × (capability_on_benchmark − benchmark_difficulty)
│ │ │ │
│ │ │ └── where this benchmark's
│ │ │ ratings are centred
│ │ │
│ │ └── same term as in the bounded formula
│ │
│ └── rating gained per unit of capability
│
└── the expected rating, standardized per benchmark (mean 0, sd 1);
no floor and no S-curve: ratings are already open-endedReasoning settings. A model config’s capability is its model’s base capability plus a reasoning offset for that setting. Some benchmarks reward reasoning more than others, so each benchmark also scales how much the offset counts on it.
capability_on_benchmark = model_base_capability + benchmark_reasoning_scale × reasoning_offset
│ │ │ │
│ │ │ └── see next block
│ │ │
│ │ └── how much this benchmark rewards reasoning
│ │ (> 1 where thinking helps most, < 1 where
│ │ it helps least; one per benchmark,
│ │ averages to 1)
│ │
│ └── the model's ability, shared by all its settings;
│ mean 0, sd 1 over models with an observed default
│ (one per model, learned)
│
└── one per model config × benchmarkThe offset is the setting’s typical effect across all models, plus how far this model departs from typical. A model with few results at a setting stays close to the typical effect.
reasoning_offset = setting_reasoning_effect + model_setting_deviation
│ │ │
│ │ └── how this model differs from typical here
│ │ (one per model × setting; pulled toward 0
│ │ when the model has few results there;
│ │ 0 for each model's pinned config)
│ │
│ └── typical boost or penalty of this setting across all
│ models: effort:high > 0, disabled < 0
│ (one per setting; 0 for "default")
│
└── one per model config; can be positive or negativeThe published score is a model config’s capability on the average benchmark, for the model’s best config.
model_config_capability = model_base_capability + reasoning_offset
│
├── equals capability_on_benchmark on an average benchmark
│ (where benchmark_reasoning_scale = 1)
│
└──→ take each model's best config
──→ rescale to the two reference models' fixed scores
──→ publish with an intervalPut together, the left branch below is fitted against benchmark scores and the right branch is published:
setting_reasoning_effect ──┐
├──→ reasoning_offset
model_setting_deviation ───┘ │
┌──────────────────┴──────────────────┐
↓ ↓
× benchmark_reasoning_scale (scale = 1)
│ │
↓ ↓
+ model_base_capability + model_base_capability
│ │
↓ ↓
capability_on_benchmark model_config_capability
│ │
− benchmark_difficulty ↓
× benchmark_steepness best config per model,
│ rescaled to the anchors
┌───────────┴───────────┐ │
↓ ↓ ↓
sigmoid, lifted predicted_rating published score
by chance_floor
│
↓
predicted_scoreUncertainty. The model is Bayesian: it produces thousands of plausible versions of every capability, consistent with the data, and the published score and interval summarize them. Sparse benchmarks borrow strength from the typical pattern across all benchmarks, so a handful of results cannot produce an extreme estimate.
How cost is estimated
Relative cost uses the same idea on reported dollars per task. Cost multiplies (price per token times tokens per task), so it is modeled on a log scale, where a reported cost is the benchmark’s cost for GPT-5 (high) plus the model config’s cost, scaled by how strongly that benchmark spreads models apart. There is no S-curve, because log cost has no floor or ceiling.
The benchmark’s line. Each benchmark has its own cost for GPT-5 (high) and its own spread.
predicted_log10_cost = benchmark_reference_cost + benchmark_cost_spread × ( … )
│ │ │ │
│ │ │ └── broken down next
│ │ │
│ │ └── how much this benchmark stretches cost
│ │ gaps between models; always > 0
│ │ (one per benchmark, averages to 1)
│ │
│ └── what GPT-5 (high) costs per task here, priced
│ the way this benchmark's evaluators price a run
│ (one per benchmark, learned)
│
└── expected cost per task in log10 dollars; reported costs scatter
around it (Student-t, noise level per evaluator), and each run
counts once however many rows report itRuns inside an agent harness get an extra offset of their own, so they don’t distort the model config’s cost. The published numbers leave it out.
( model_config_cost + harness_offset )
│ │
│ └── extra cost of running inside an agent harness,
│ best-of-n or self-refine loop; 0 without one
│ (one per harnessed config; not published)
│
└── this model config's cost relative to GPT-5 (high), in log10
units: 0 = same cost, 1 = ten times, −0.3 = half
(broken down below)Reasoning settings. A model config’s cost is its model’s base cost, plus the setting’s typical cost across all models, plus how far this model departs from typical. Higher effort always costs more on average; a model whose efforts spread wider or narrower than typical shows it through its own deviation.
model_config_cost = model_base_cost + setting_cost_effect + config_cost_deviation
│ │ │ │
│ │ │ └── how this model's cost here
│ │ │ departs from typical (one per
│ │ │ model config; 0 for each model's
│ │ │ pinned config: its default, else
│ │ │ its most-reported)
│ │ │
│ │ └── typical cost of this setting across all models;
│ │ effort climbs a ladder, minimal < low < … < max;
│ │ other settings get a free value; 0 for "default"
│ │ (one per setting, learned)
│ │
│ └── the model's own cost level, shared by all its settings
│ (one per model, learned)
│
└── one per model config, measured from GPT-5 (high), which is fixed at 0The published numbers are the relative cost, a multiple of GPT-5 (high)’s cost free of any benchmark mix, and typical dollars per task on the two workloads.
cost_index = 10 ^ model_config_cost
│
└── times what GPT-5 (high) costs: 0.5 = half, 3 = three times
(median with an interval)
workload_usd = 10 ^ mean over the workload's benchmarks of
(benchmark_reference_cost + benchmark_cost_spread × model_config_cost)
│
└── no harness_offset, so dollars as those benchmarks' own evaluators would report
them; the agentic benchmarks only ever run inside a harness, so their
benchmark_reference_cost already includes a typical onePut together, the left branch below is fitted against reported costs and the two right branches are published:
model_base_cost ─────────┐
setting_cost_effect ─────┼──→ model_config_cost (GPT-5 (high) = 0)
config_cost_deviation ───┘ │
┌────────────────┴──────────────────┬─────────────────────┐
↓ ↓ ↓
+ harness_offset 10 ^ (…) workload benchmarks,
│ │ no harness_offset
↓ ↓ │
× benchmark_cost_spread cost_index ↓
│ workload_usd
↓
+ benchmark_reference_cost
│
↓
predicted_log10_costWorkload dollars are what the evaluators who run those benchmarks would report for a run, in the setup they run it in. They are not converted to a common pricing convention, so they inherit those evaluators’ choices, such as which prices they use.
How well it holds up
A model can fit its data well and still predict badly, so Atlas is tested on benchmarks it never saw. Benchmarks are set aside in folds, every benchmark from one evaluator in the same fold, the model is refit without them or anything from the same evaluator or of the same benchmark (another evaluator’s run, another version), and its ranking of models is compared with the actual results on the set-aside benchmarks (Spearman rank agreement, ρ: 1 is a perfect ranking). Other published indices are scored the same way on the same benchmarks, using only benchmarks they were not built from either.
The held-out comparison has not been run.
Limitations
- Coverage is uneven and not random. Only a small share of model-benchmark pairs are observed, and stronger models are tested on more benchmarks (Figure 1). The model assumes that which benchmarks a model was run on carries no extra information once its capability is known.
- One dimension. Capability summarizes the shared pattern across benchmarks. Benchmark-specific strengths remain, and capability says nothing about safety, reliability or product quality.
- Reasoning labels are nominal. One lab’s “high” effort may be another’s “medium”. Per-model terms absorb some of this, but the labels are not compute-calibrated.
- Harnesses are avoided, not modeled. Each benchmark is read under its most common setup, but a model reported only under another setup is still compared on the same curve, so some harness advantage can enter capability.
- Relative units. Scores depend on the models covered and on the two reference models. Cost dollars depend on how the evaluators who report them price runs.
- Public data. Source metadata can be wrong or incomplete. Conflicts are audited and dropped, but curation cannot catch every error.
Technical details
›Preparing the data
- Direction. Only benchmarks with a declared direction are used; lower-is-better scores are reflected so that higher always means better.
- Duplicates. Scores whose sampling protocol (single attempt or consensus) does not match the benchmark’s usual one are dropped. Reports that agree within twice the sampling noise expected between two runs collapse to their median. Averaged runs beat conflicting single attempts, a lone outlier among three or more reports is removed, and any remaining conflict is dropped and logged. Repetition earns no extra weight.
- Guards. A score below half the guessing floor is dropped, since guessing alone would clear it; it signals refusals or unparseable answers. No-thinking harness rows are kept only for models with no reasoning mode. A cell dropped as a conflict cannot be filled from another configuration or benchmark.
- Benchmark families. Each benchmark declares a family and a role (primary, component, diagnostic). A family uses its primary benchmark if that alone reaches the coverage threshold, otherwise its single component; diagnostics are never used.
- One setup per benchmark. Configurations of one model, setting and benchmark are pooled under the setup with the most models; provider routes of one run collapse to their median.
- Scale. Bounded scores are placed on their declared range, 0 to 1. Ratings with no range (Elo, TrueSkill, Bradley–Terry) are standardized per family.
- Coverage. Repeated until stable: a family needs a minimum number of models, a model needs a minimum number of families across its states, and a family’s scores must vary.
›The capability likelihood
The full curve is , where is chance_floor, benchmark_steepness (in item-response terms, its discrimination), benchmark_difficulty, model_base_capability, reasoning_offset and benchmark_reasoning_scale (). The published capability is (model_config_capability), capability on the average benchmark.
is a shared effect per reasoning setting plus a free deviation for each model at that setting, shrunk toward zero with a per-setting scale.
- Bounded scores use a Beta with a per-benchmark dispersion drawn from a shared population. The dispersion covers everything that moves a score between runs, including which questions a benchmark happens to contain, so a short benchmark is not read as more precise than its questions allow.
- Exact zeros and perfect scores get their own probabilities, which rise the further a model sits below (or above) the benchmark, so refusals and harness failures do not distort the curve.
- Ratings are Normal around , with no floor.
›Priors and identification
Benchmark difficulties, discriminations and dispersions are partially pooled: each is drawn from a population whose centre and spread are learned, so sparse benchmarks are pulled toward the typical pattern. Reasoning effects have hierarchical scales.
Three constraints fix the scale. Base capability is standardized to mean 0 and SD 1 across models with an observed default state. One state per model (its default where observed) has no deviation of its own. The log reasoning scales sum to zero. The two reference models then convert this scale to score points in every posterior draw, so the intervals include the uncertainty of the scale itself.
›Cost model details
- One run counts once. Evaluators often report one run many times (a summary next to its sub-scores, or one cost copied onto every metric). Such rows are grouped into one run and weighted 1/rows.
- Priors.
benchmark_reference_costis within its suite (a suite and its sub-scores), with , so one run of a suite counts once however many sub-scores it reports;benchmark_cost_spreadis , nested in its suite the same way, with summing to zero; the effort ladder insetting_cost_effectrises monotonically;config_cost_deviationandharness_offsetare zero-mean with learned scales, and one state per model is pinned. - Noise is Student-t with a scale per evaluator, so occasional outliers do not drag a state.
- No evaluator offsets. Evaluators price runs differently, but an offset per evaluator made held-out predictions no better; each benchmark’s level carries its evaluators’ pricing instead.
- API defaults. A run with no effort named, on a model whose API documents its default effort, is priced at that effort.
›Sampling and diagnostics
Both models are fitted with nutpie’s No-U-Turn sampler. Difficulty is sampled in a shifted coordinate that removes a ridge between difficulty and discrimination without changing the posterior.
›Alternatives that were tested
Each part of both models was kept because removing it made held-out predictions worse, and each alternative below was dropped because it did not help. The measurements are recorded in the model code.
- Capability: a second latent dimension (not identifiable), no guessing floor (same rankings, but inflated scores for weak models and flattened curves on multiple-choice benchmarks), an asymmetric curve, a plain Beta likelihood, per-model curves along the effort ladder.
- Cost: evaluator offsets, per-model stretch of the effort ladder, and harness effects shared across models.
›Reproducing the fits
The capability model is model/fit.py, with its data preparation in model/prepare.py; the cost model is model/cost.py. Each file documents its design and the tests behind it.
uv run -m model.refit # refit both models and publish the scores
uv run -m model.validate # rerun the held-out tests (slow)Every published fit is stamped with a hash of its input data, and publishing refuses a fit made from other data.