Atlas

Benchmarks

AutoRLBench v2 (issue level)

GitHub

AutoRLBench's second task list: 32 real reinforcement-learning training runs that went wrong, across six failure categories (configuration, reward, data, environment, algorithm and infrastructure). At the issue level an agent reads the run's code, configs, logs and metrics and writes a diagnosis, which an LLM judge compares with a hidden oracle; a task passes only with a full verdict and root-cause and evidence scores above 0.8. Each model runs each task once.

Models
6
Top score
46.9
Leader
GPT-6 Astra
Updated
—
Resolution
—
Saturation
—

No signal or difficulty estimate. Held out of the fit by design.

6 dated models · 3 frontier modelsClosed weightsOpen weights
Data: AutoRLBench Authorsatlas.village.dev
Loading Atlas data…