Benchmarks
347 sources · 1,258 models × 3,038 benchmarks · 158,912 scoresUniPat's SPU tests whether an agent can predict the results of real experiments before running them. Each task comes from one wet-lab study published in a Nature Portfolio journal between January 2025 and August 2026: the agent ranks the candidate experiments in each comparison group by a hidden measured outcome, such as yield, conductivity or binding, with eight hours, one GPU and web access through audited tools that end the run if it finds the source study. An offline grader scores each group by Spearman correlation with the measurements and each task as a fixed-weight sum of its groups, so a task scores from −1 to 1 and a random valid ranking averages 0. The 100 public tasks (85 papers; chemistry, materials science, biology and medicine) were each run once per configuration in Claude Code or Codex. The score is the grader's default mean over all 100 official runs, in which a refusal, a missing file or a malformed file scores −1, so every configuration shares the same denominator.
475th strongest of 798 fitted benchmarks
Signal-to-noise +4.84
17th hardest of 798 fitted benchmarks
Capability needed ≳ 189.7
| # | Model | Setting | Score | Source |
|---|---|---|---|---|
| 1 | GPT-6 Astra | medium · Codex | 0.4 | |
| 2 | Claude Sonnet 5.5 | max · Claude Code | 0.3 | |
| 3 | GPT-6.1 Sol | max · Codex | 0.3 | |
| 4 | DeepSeek V4.1 Flash | max · Claude Code | 0.2 | |
| 5 | Kimi K3.1 Pro | max · Claude Code | 0.2 | |
| 6 | Claude Opus 5 | max · Claude Code | 0.2 | |
| 7 | Qwen3.8 Max (0902) | max · Claude Code | 0.1 | |
| 8 | GLM-5.3 Flash | max · Claude Code | 0.0 | |
| 9 | Claude Opus 5.5 | max · Claude Code | -0.3 | |
| 10 | Claude Fable 5.1 | max · Claude Code | -0.5 |
Benchmarks outside its family whose model rankings track it most reliably (Spearman ρ over at least 10 shared models).