Benchmarks
357 sources · 1,263 models × 3,079 benchmarks · 162,127 scoresHow often the 95% intervals agents report in SciResearch-Eval contain the exact true property, pooled over the three properties of the 38 held-out tasks whose simulator is missing physics and over every run, with an unsubmitted run counting as three misses. A calibrated agent scores about 95; far below is confidently wrong, and above 95 is cautious rather than better, since uselessly wide intervals reach 100. The target is therefore 95, and the interval score, not coverage, ranks models. A diagnostic of the interval score.
No signal or difficulty estimate. Held out of the fit by design.
| # | Model | Setting | Score | Source |
|---|---|---|---|---|
| 1 | GPT-6 Astra | not reported | 93.6 | |
| 2 | Claude Opus 5.5 | max | 93.0 | |
| 3 | Claude Fable 5.2 | max | 98.2 | |
| 4 | Claude Haiku 5.5 | max | 90.1 | |
| 5 | Claude Sonnet 5.5 | max | 89.5 | |
| 6 | Sonnet 4.5 | 32k | 44.7 | |
| 7 | Qwen3.8 27B | thinking | 38.3 | |
| 8 | Gemma 4 31B IT | disabled | 21.9 | |
| 9 | Muse Glimmer 30B | high | 20.5 |