← All benchmarks

AutoResearchExam (hidden-test AUARC)

Agents · 2026-09-09

Bespoke Labs' AutoResearchExam gives an agent 29 open-ended machine learning research tasks (22 on CPU, 7 on GPU), such as faster CPU language-model decoding, group-robust coreset selection, MILP branching or sparse autoencoder dictionary learning, each with a 24-hour research window in the same Terminus 2 based harness and no internet access. The agent submits as often as it likes and sees only validation scores; each improvement of its best validation score is also graded on a sealed hidden test set. Raw task metrics are mapped to a 0-1 reward (a trivial baseline scores 0 and a strong reference about 0.5), and hidden-test AUARC (area under the AutoResearch curve) is the time-weighted average of the hidden-test reward held over the run, so early gains count for more; the score is its mean over the tasks, from one run per task. It differs from FrontierSWE, PostTrainBench and FrontierML in its evaluator, harness and anytime scoring.

Top models (higher is better)