CritPt v2 (agentic, avg@5)
Science
CritPt v2 is the CritPt authors' revision of the dataset: its 70 research-level physics challenges, reviewed with physics expert collaborators to correct underspecified statements, errors and typos. Artificial Analysis runs it agentically in its Stirrup harness, in an offline Linux sandbox with Python 3.13 (numpy, scipy, sympy, matplotlib) and at most 50 turns, using the authors' one-step prompt for models with code tools; the authors' grading server scores each repeat, and the reported pass@1 is the mean over five repeats per challenge. Scores are not comparable with the tool-free CritPt v1.
Top models (higher is better)
| Model | Score |
|---|---|
| GPT-6 Astra | 83.7 |
| GPT-6.1 Sol | 82.0 |
| GPT-6.1 Terra | 79.4 |
| Claude Opus 5.5 | 78.9 |
| Gemini 4 Carbon | 78.9 |
| Claude Fable 5.1 | 76.9 |
| GPT-5.6 Sol | 74.9 |
| Claude Sonnet 5.5 | 74.0 |
| Muse Wonder Full | 73.1 |
| Gemini 4 Argon | 72.9 |
| Muse Wonder | 69.7 |
| Kimi K3.1 Pro | 68.0 |
| Muse Spark 1.3 | 66.9 |
| Claude Haiku 5.5 | 64.6 |
| GPT-6 Luna | 64.6 |
| Gemini 3.8 Flash | 64.0 |
| Kimi K3 | 61.4 |
| Grok 4.7 | 60.9 |
| GLM-5.3 | 57.4 |
| Ling 3.1 Flash 1019 | 57.1 |
| MiMo-V2.6-Pro | 56.9 |
| Step 5 Preview | 54.6 |
| Muse Spark Open | 51.4 |
| Qwen3.8 Max (0902) | 50.9 |
| MiniMax M3.1 Flash | 50.6 |
| MiMo-V2.6-Flash | 50.3 |
| DeepSeek V4.1 Flash 1017 | 50.0 |
| DeepSeek V4 Pro 0813 | 45.7 |
| MiniMax M3 | 44.6 |
| Mistral Large 4 Preview | 43.4 |
| Beam 1.0 501B A23B | 40.0 |
| Nemotron 3.5 Super VL 120B A12B (precision unspecified) | 36.9 |
| Inkling-Small | 34.0 |
| K2 Horizon 375B A23B | 29.7 |
| Inkling | 29.4 |
| GPT-5 | 28.9 |
| Nemotron 3 Ultra 550B A55B | 28.9 |
| Muse Glimmer 30B | 24.3 |
| Qwen3.8 27B | 24.0 |
| Gemini 3.5 Flash-Lite | 16.0 |