BrokenArXiv 09/2026 (Native Harnesses)
Math
MathArena's September 2026 false-statement track has 53 questions, run under the August native-harness protocol with Python and SageMath and no internet and a 12-hour and $100 limit per problem. Gemini 3.8 Flash awards 0–3 points per question, reserving full credit for identifying falsity, and scores normalize mean points to 0–100.
Top models (higher is better)
| Model | Score |
|---|---|
| GPT-6.1 Sol | 82.7 |
| Claude Opus 5.5 | 76.7 |
| DeepSeek V4.1 Flash | 54.7 |
| Gemini 4 Argon | 49.7 |
| Grok 4.7 | 27.7 |
| Gemini 3.8 Flash | 26.7 |