← All benchmarks
ScarfBench All Tests Pass
Code · 2026-10-27
Share of the 204 ScarfBench Java migrations whose behavioral smoke-test suite passes completely, as IBM's All pass % reports it. IBM's post states no denominator, and some published values aren't whole numbers of tasks over 204.
Benchmark source: Ibm
Top models (higher is better)
| Model | Score | Source |
|---|
| Claude Opus 5.5 | 72.1 | IBM Research ScarfBench post, October 27, 2026 (Claude Opus 5.5 (Claude Code)) · IBM |
| Gemini 4 Argon | 71.1 | IBM Research ScarfBench post, October 27, 2026 (Gemini 4 Argon (Antigravity)) · IBM |
| GPT-6.1 Sol | 62.7 | IBM Research ScarfBench post, October 27, 2026 (GPT-6.1 Sol (Codex)) · IBM |
| Gemini 3.8 Flash | 62.3 | IBM Research ScarfBench post, October 27, 2026 (Gemini 3.8 Flash (mini-swe)) · IBM |
| GPT-6 Astra | 61.4 | IBM Research ScarfBench post, October 27, 2026 (GPT-6 Astra (Codex)) · IBM |
| Claude Opus 5 | 60.8 | IBM Research ScarfBench post, October 27, 2026 (Claude Opus 5 (Claude Code)) · IBM |
| Kimi K3 | 50.7 | IBM Research ScarfBench post, October 27, 2026 (Kimi K3 (kimi-code)) · IBM |
| Gemini 3.1 Pro Preview | 35.1 | IBM Research ScarfBench post, October 27, 2026 (Gemini 3.1 Pro (OpenHands)) · IBM |