Benchmarks
347 sources · 1,258 models × 3,038 benchmarks · 158,912 scoresMathArena's October 2026 false-statement track has 54 questions, run under the August native-harness protocol with Python and SageMath and no internet and a 12-hour and $100 limit per problem. Gemini 3.8 Flash awards 0–3 points per question, reserving full credit for identifying falsity, and scores normalize mean points to 0–100.
401st strongest of 798 fitted benchmarks
Signal-to-noise +5.80 for the BrokenArXiv — Overall family
Another benchmark represents this family in the fit.
| # | Model | Setting | Score | Source |
|---|---|---|---|---|
| 1 | Gemini 4 Carbon | high · MathArena Antigravity CLI 1.5.5; Docker with Python and SageMath; no internet | 82.7 | |
| 2 | GPT-6.1 Sol | max · MathArena Codex; Docker with Python and SageMath; no internet | 80.3 | |
| 3 | Claude Opus 5.5 | high · MathArena native CLI evaluation; Docker with Python and SageMath; no internet; exact CLI implementation and version unreported | 77.8 | |
| 4 | Claude Fable 5.2 | max · MathArena Claude Code 2.1.321; Docker with Python and SageMath; no internet; OpenRouter restricted to Anthropic; 128000 output-token environment setting | 73.5 | |
| 5 | Inkling 2 Small | 0.99 · MathArena OpenCode 1.19.5; Tinker API; Docker with Python and SageMath; no internet; temperature 1.0 | 63.3 | |
| 6 | Kimi K3.1 Pro | max · MathArena Kimi Code 2.3.1; Docker with Python and SageMath; no internet; OpenRouter pinned to MoonshotAI's mxfp4 endpoint without provider fallback | 58.0 | |
| 7 | DeepSeek V4.1 Flash 1017 | max · DeepSeek Harness | 49.7 | |
| 8 | Gemini 4 Argon | high · MathArena Antigravity CLI 1.3.5; Docker with Python and SageMath; no internet | 39.8 | |
| 9 | Muse Spark Open | x-high · MathArena Muse CLI 1.4.9-R6297.1; Docker with Python and SageMath; no internet; OpenRouter pinned to Together without provider fallback | 33.3 |