ErdosBench
Math · 2026-06-08
Ulam's research-mathematics leaderboard on the 226 private Erdős-inspired candidate problems of its v0.1 corpus, scoring finding decisive obstructions, using known theorems correctly, scoped partial progress and avoiding unsafe solved claims. Each run answers every problem once, through APIs in a non-agentic setup with a 100,000-token limit per problem (GPT models through Codex and Claude Opus 4.8 through Claude Code), and an external judge that Ulam does not name grades each response A (sound and scoped), B (useful but incomplete), C (material gap), D (likely wrong), F (contradicted) or M (missing). The score is the published judge average on a 0-4 scale over all 226 problems, with missing rows scored 0; problems, answer keys and grades are private.
Top models (higher is better)
| Model | Score |
|---|---|
| Claude Fable 5.1 | 3.3 |
| Claude Opus 5.5 | 3.2 |
| GPT-6 Astra | 3.2 |
| GPT-5.6 Sol | 3.1 |
| Muse Wonder | 3.0 |
| DeepSeek V4.1 Flash 1017 | 3.0 |
| Gemini 4 Argon | 3.0 |
| MiMo-V2.6-Pro | 3.0 |
| MiniMax M3.1 Flash | 3.0 |
| Kimi K3.1 Pro | 3.0 |
| GLM-5.3 Flash | 3.0 |
| Hy4 Preview | 2.9 |
| Muse Spark 1.3 | 2.9 |
| Qwen3.8 27B | 2.8 |
| Kimi K3 | 2.7 |
| Muse Spark 1.2 | 2.7 |
| GPT-5.5 | 2.7 |
| Kimi K2.7 Code | 2.6 |
| Opus 4.8 | 2.5 |
| GLM-5.2 | 2.4 |
| Mistral Large 4 Preview | 2.4 |
| MiniMax M3 | 2.1 |