PaperBench Code-Dev
Science · 2025-04-02
PaperBench Code-Dev is the lighter variant of OpenAI's PaperBench: an agent must write a codebase that replicates one of 20 ICML 2024 Spotlight and Oral papers, and an LLM judge grades only the code-development leaves of each paper's author-approved rubric, without executing the code or checking results. Scores report the mean rubric replication score in percent. Higher is better.
Top models (higher is better)
| Model | Score |
|---|---|
| Kimi K2.5 | 63.5 |
| Kimi K2 Instruct | 27.8 |