Benchmarks
347 sources · 1,258 models × 3,038 benchmarks · 158,912 scoresProphet Arena (UChicago SIGMA Lab with Kalshi, arXiv 2510.17638) has models forecast open Kalshi prediction-market events and scores them once the events resolve; in the agentic track each model runs as an agent in an OpenCode harness and gathers its own information, with up to an hour per event. The Arena score is each forecaster's skill from an item-response fit over every resolved event and forecaster, anchored so the market's own prices sit at zero skill, translated to predicted 1 - Brier on one shared reference set and shown x100. The fit is joint across the cohort and refit as events resolve, so every value moves with the cohort and the score is diagnostic.
No signal or difficulty estimate. Held out of the fit by design.
| # | Model | Setting | Score | Source |
|---|---|---|---|---|
| 1 | GPT-6 Astra | not reported · OpenCode | 86.2 | |
| 2 | Claude Opus 5.5 | not reported · OpenCode | 86.2 | |
| 3 | Kimi K3.1 Pro | not reported · OpenCode | 86.1 | |
| 4 | Gemini 4 Argon | not reported · OpenCode | 86.0 | |
| 5 | Claude Fable 5.1 | not reported · OpenCode | 86.0 | |
| 6 | GPT-6.1 Sol | not reported · OpenCode | 86.0 | |
| 7 | Gemini 3.8 Flash | not reported · OpenCode | 86.0 | |
| 8 | Claude Haiku 5.5 | not reported · OpenCode | 85.9 | |
| 9 | GLM-5.3 | not reported · OpenCode | 85.7 | |
| 10 | Claude Sonnet 5.5 | not reported · OpenCode | 85.7 | |
| 11 | GPT-6.1 Terra | not reported · OpenCode | 85.7 | |
| 12 | GPT-6 Luna | not reported · OpenCode | 85.6 | |
| 13 | Grok 4.6 | not reported · OpenCode | 85.6 | |
| 14 | Gemini 3.1 Pro Preview | not reported · OpenCode | 85.5 | |
| 15 | Muse Wonder | not reported · OpenCode | 85.4 | |
| 16 | Kimi K3 | not reported · OpenCode | 85.3 | |
| 17 | GLM-5.2 | not reported · OpenCode | 85.3 |
Benchmarks outside its family whose model rankings track it most reliably (Spearman ρ over at least 10 shared models).