Atlas

Benchmarks

BrokenArXiv 10/2026 (Native Harnesses)

MathArena's October 2026 false-statement track has 54 questions, run under the August native-harness protocol with Python and SageMath and no internet and a 12-hour and $100 limit per problem. Gemini 3.8 Flash awards 0–3 points per question, reserving full credit for identifying falsity, and scores normalize mean points to 0–100.

Models
9
Top score
82.7
Leader
Gemini 4 Carbon
Updated
—
Resolution
—
Saturation
—
Signal
50thpercentile
195 benchmarks from 0.8 to 3.5224 benchmarks from 3.5 to 6.1133 benchmarks from 6.1 to 8.898 benchmarks from 8.8 to 11.566 benchmarks from 11.5 to 14.126 benchmarks from 14.1 to 16.815 benchmarks from 16.8 to 19.59 benchmarks from 19.5 to 22.18 benchmarks from 22.1 to 24.84 benchmarks from 24.8 to 27.55 benchmarks from 27.5 to 30.12 benchmarks from 30.1 to 32.84 benchmarks from 32.8 to 35.45 benchmarks from 35.4 to 38.11 benchmark from 38.1 to 40.82 benchmarks from 43.4 to 46.11 benchmark from 62.1 to 64.8

401st strongest of 798

Signal-to-noise +5.80 for the BrokenArXiv — Overall family

Difficulty
Not estimated

Another benchmark represents this family in the fit.

Organizations
9 dated models · 3 frontier modelsClosed weightsOpen weights
Data: MathArenaatlas.village.dev

BrokenArXiv 10/2026 (Native Harnesses) leaderboard

#ModelSettingScoreSource
1Gemini 4 Carbonhigh · MathArena Antigravity CLI 1.5.5; Docker with Python and SageMath; no internet
82.7
2GPT-6.1 Solmax · MathArena Codex; Docker with Python and SageMath; no internet
80.3
3Claude Opus 5.5high · MathArena native CLI evaluation; Docker with Python and SageMath; no internet; exact CLI implementation and version unreported
77.8
4Claude Fable 5.2max · MathArena Claude Code 2.1.321; Docker with Python and SageMath; no internet; OpenRouter restricted to Anthropic; 128000 output-token environment setting
73.5
5Inkling 2 Small0.99 · MathArena OpenCode 1.19.5; Tinker API; Docker with Python and SageMath; no internet; temperature 1.0
63.3
6Kimi K3.1 Promax · MathArena Kimi Code 2.3.1; Docker with Python and SageMath; no internet; OpenRouter pinned to MoonshotAI's mxfp4 endpoint without provider fallback
58.0
7DeepSeek V4.1 Flash 1017max · DeepSeek Harness
49.7
8Gemini 4 Argonhigh · MathArena Antigravity CLI 1.3.5; Docker with Python and SageMath; no internet
39.8
9Muse Spark Openx-high · MathArena Muse CLI 1.4.9-R6297.1; Docker with Python and SageMath; no internet; OpenRouter pinned to Together without provider fallback
33.3

Related benchmarks

Loading Atlas data…