BrokenArXiv 06/2026
Math
This MathArena track evaluates whether models recognize and refuse to prove 54 plausible but false mathematical statements extracted from recent arXiv papers. Scores report the percentage of problems answered correctly.
Top models (higher is better)
| Model | Score |
|---|---|
| Claude Opus 5.5 | 100.0 |
| GPT-6.1 Sol | 100.0 |
| GPT-6 Astra | 99.1 |
| GPT-6 Sol | 95.4 |
| Claude Opus 5 | 90.7 |
| Claude Fable 5.1 | 86.6 |
| GPT-5.5 | 69.4 |
| GPT-5.6 Sol | 67.3 |
| Muse Spark 1.3 | 53.2 |
| Grok 4.7 | 52.8 |
| Kimi K3 | 51.9 |
| Muse Spark 1.1 | 50.6 |
| Claude Fable 5 | 47.8 |
| Opus 4.8 | 41.7 |
| DeepSeek V4.1 Flash | 39.4 |
| Grok 4.5 | 29.3 |
| Mistral Large 4 Preview | 22.2 |
| Gemini 3.8 Flash | 20.8 |
| Gemini 3.1 Pro Preview | 17.6 |
| DeepSeek-V4-Flash | 16.7 |
| Step 3.7 Flash | 15.7 |
| GLM-5.2 | 14.2 |
| Gemini 3.5 Flash | 12.3 |
| Gemini 3.6 Flash | 12.0 |
| Gemini 3.7 Flash | 11.1 |
| Qwen3.6 35B A3B | 3.4 |
| Qwen3.5-2B | 2.8 |