PaperBench-10 - OpenAI pass@1
Code · 2025-08-07
OpenAI's 10-paper subset of PaperBench (papers needing under 10GB of external data), in which an agent must replicate an ICML 2024 Spotlight or Oral paper from scratch and is graded against hierarchical replication rubrics. Reported by OpenAI as pass@1 without browsing. Higher is better.
Top models (higher is better)
| Model | Score |
|---|---|
| GPT-5.2-Codex | 43.0 |
| GPT-5.1-Codex-Max | 40.0 |
| GPT-5.2 | 39.0 |
| GPT-5.1 | 34.0 |
| GPT-5 | 24.0 |