Benchmarks
326 sources · 1,256 models × 2,956 benchmarks · 154,484 scoresAutoRLBench's second task list: 32 real reinforcement-learning training runs that went wrong, across six failure categories (configuration, reward, data, environment, algorithm and infrastructure). At the issue level an agent reads the run's code, configs, logs and metrics and writes a diagnosis, which an LLM judge compares with a hidden oracle; a task passes only with a full verdict and root-cause and evidence scores above 0.8. Each model runs each task once.
No signal or difficulty estimate. Held out of the fit by design.
| # | Model | Setting | Score | Source |
|---|---|---|---|---|
| 1 | GPT-6 Astra | max · Codex | 46.9 | |
| 2 | GPT-6.1 Sol | max · Codex | 43.8 | |
| 3 | GLM-5.3 | max · Claude Code | 31.3 | |
| 4 | Claude Fable 5.1 | max · Claude Code | 25.0 | |
| 5 | Kimi K3.1 Pro | max · Claude Code | 25.0 | |
| 6 | Kimi K3 | max · Claude Code | 0.0 |