OpenAI PRs - Preparedness pass@1
Code · 2025-08-07
OpenAI Preparedness evaluation in which an agent, checked out to the pre-PR branch of an internal OpenAI repository, must reproduce a real pull request by an OpenAI employee from a human-written prompt; a rollout passes only if all hidden task-specific unit tests pass. Reported as pass@1 without browsing. From the GPT-5.3-Codex card onward OpenAI calls it Monorepo-Bench (pull-request style contributions in a large internal repository, graded by hidden tests), with unchanged comparison values. Task set is internal; diagnostic. Higher is better.
Top models (higher is better)
| Model | Score |
|---|---|
| GPT-5.5 | 60.0 |
| GPT-5.4 | 59.3 |
| GPT-5.2 | 56.7 |
| GPT-5.3-Codex | 56.0 |
| GPT-5.4 Mini | 54.0 |
| GPT-5.1-Codex-Max | 53.0 |
| GPT-5 | 45.0 |
| GPT-5.1 | 45.0 |