← All benchmarks
ScarfBench Mean
Code · 2026-10-27
The Mean % column of IBM's ScarfBench table over 204 Java migrations. IBM's post doesn't define it. Its values sit above the suite-weighted smoke score, as an unweighted mean across tasks would, but that reading isn't stated.
Benchmark source: Ibm
Top models (higher is better)
| Model | Score | Source |
|---|
| Gemini 4 Argon | 85.9 | IBM Research ScarfBench post, October 27, 2026 (Gemini 4 Argon (Antigravity)) · IBM |
| Claude Opus 5.5 | 82.0 | IBM Research ScarfBench post, October 27, 2026 (Claude Opus 5.5 (Claude Code)) · IBM |
| Claude Opus 5 | 79.7 | IBM Research ScarfBench post, October 27, 2026 (Claude Opus 5 (Claude Code)) · IBM |
| Gemini 3.8 Flash | 77.6 | IBM Research ScarfBench post, October 27, 2026 (Gemini 3.8 Flash (mini-swe)) · IBM |
| GPT-6.1 Sol | 76.8 | IBM Research ScarfBench post, October 27, 2026 (GPT-6.1 Sol (Codex)) · IBM |
| GPT-6 Astra | 75.4 | IBM Research ScarfBench post, October 27, 2026 (GPT-6 Astra (Codex)) · IBM |
| Kimi K3 | 71.2 | IBM Research ScarfBench post, October 27, 2026 (Kimi K3 (kimi-code)) · IBM |
| Gemini 3.1 Pro Preview | 50.5 | IBM Research ScarfBench post, October 27, 2026 (Gemini 3.1 Pro (OpenHands)) · IBM |