Benchmarks
347 sources · 1,258 models × 3,038 benchmarks · 158,912 scoresProphet Arena (UChicago SIGMA Lab with Kalshi, arXiv 2510.17638) has models forecast open Kalshi prediction-market events and scores them once the events resolve; in the Fixed Context track each model makes one call on a centrally curated research context, with no web search or tools. The Arena score is each forecaster's skill from an item-response fit over every resolved event and forecaster, anchored so the market's own prices sit at zero skill, translated to predicted 1 - Brier on one shared reference set and shown x100. The fit is joint across the cohort and refit as events resolve, so every value moves with the cohort and the score is diagnostic.
No signal or difficulty estimate. Held out of the fit by design.
| # | Model | Setting | Score | Source |
|---|---|---|---|---|
| 1 | Claude Opus 5.5 | not reported | 85.5 | |
| 2 | Kimi K3.1 Pro | not reported | 85.4 | |
| 3 | Claude Haiku 5.5 | not reported | 85.4 | |
| 4 | Gemini 4 Argon | not reported | 85.3 | |
| 5 | Claude Sonnet 5.5 | not reported | 85.3 | |
| 6 | Gemini 3.8 Flash | not reported | 85.3 | |
| 7 | GPT-6.1 Terra | not reported | 85.2 | |
| 8 | Muse Wonder | not reported | 85.2 | |
| 9 | Claude Fable 5.1 | not reported | 85.0 | |
| 10 | Grok 4.6 | not reported | 85.0 | |
| 11 | Claude Fable 5 | not reported | 84.9 | |
| 12 | Gemini 3.1 Pro Preview | not reported | 84.9 | |
| 13 | GPT-5.6 Sol | not reported | 84.8 | |
| 14 | GPT-6.1 Sol | not reported | 84.8 | |
| 15 | GPT-6 Luna | not reported | 84.8 | |
| 16 | GPT-6 Astra | not reported | 84.7 | |
| 17 | DeepSeek V4 Pro 0813 | not reported | 84.6 | |
| 18 | GLM-5.3 | not reported | 84.6 | |
| 19 | Kimi K3 | not reported | 84.6 | |
| 20 | Hy3 | not reported | 84.6 |
Benchmarks outside its family whose model rankings track it most reliably (Spearman ρ over at least 10 shared models).