Benchmarks
357 sources · 1,263 models × 3,079 benchmarks · 162,127 scoresSynthetic Sciences' FrontierML gives an agent 25 open-ended empirical ML research problems built around active research directions, such as credit assignment under sparse terminal rewards, pretraining a language model without backpropagation or unlearning that survives fine-tuning. Each run is 12 hours in Lab0, a virtual lab with H100s rented per job from a shared cluster, a $50-125 wallet, a frozen paper library, a pinned dataset mirror and model APIs, but no validation score, grader or internet; the sealed model, method or recipe it hands in is graded on protected data or at a scale the agent could not afford. Each task's metric is normalized between a starting reference (0) and a strong-result reference set by judgment (1) and clipped to 0-1, and the score is the mean over all 25 tasks of one run each; 15 tasks are released and 10 stay private. Every model runs in Synthetic Sciences' Horizon harness at its highest reasoning effort. Unrelated to Induction Labs' FrontierML (frontierml).
No signal or difficulty estimate. Too few models have been measured on this benchmark to estimate its difficulty.
| # | Model | Setting | Score | Source |
|---|---|---|---|---|
| 1 | Claude Fable 5.2 | max · Horizon 0.4.17 (Synthetic Sciences' FrontierML build of mini-swe-agent, under Harbor) | 0.5 | |
| 2 | Claude Opus 5.5 | max · Horizon 0.4.17 (Synthetic Sciences' FrontierML build of mini-swe-agent, under Harbor) | 0.4 | |
| 3 | GPT-6 Astra | max · Horizon 0.4.17 (Synthetic Sciences' FrontierML build of mini-swe-agent, under Harbor) | 0.4 | |
| 4 | Gemini 4 Carbon | high · Horizon 0.4.17 (Synthetic Sciences' FrontierML build of mini-swe-agent, under Harbor) | 0.4 |