← All benchmarks
ScarfBench Smoke Score
Code · 2026-03-11
IBM's 204 directed migrations of 34 enterprise Java applications between Jakarta EE, Quarkus and Spring. A coding agent gets a working application and a target framework, and the result must compile, deploy and pass behavioral smoke tests written from Gherkin scenarios. The smoke score, IBM's leaderboard metric, gives partial credit and weights larger test suites more heavily. Scores are for a model paired with an agent harness, not for the model alone.
Benchmark source: Ibm
Top models (higher is better)
| Model | Score | Source |
|---|
| Gemini 4 Argon | 75.2 | IBM Research ScarfBench post, October 27, 2026 (Gemini 4 Argon (Antigravity)) · IBM |
| Claude Opus 5.5 | 70.5 | IBM Research ScarfBench post, October 27, 2026 (Claude Opus 5.5 (Claude Code)) · IBM |
| Claude Opus 5 | 69.8 | IBM Research ScarfBench post, October 27, 2026 (Claude Opus 5 (Claude Code)) · IBM |
| GPT-6.1 Sol | 68.4 | IBM Research ScarfBench post, October 27, 2026 (GPT-6.1 Sol (Codex)) · IBM |
| Gemini 3.8 Flash | 66.0 | IBM Research ScarfBench post, October 27, 2026 (Gemini 3.8 Flash (mini-swe)) · IBM |
| GPT-6 Astra | 65.4 | IBM Research ScarfBench post, October 27, 2026 (GPT-6 Astra (Codex)) · IBM |
| Kimi K3 | 61.1 | IBM Research ScarfBench post, October 27, 2026 (Kimi K3 (kimi-code)) · IBM |
| Gemini 3.1 Pro Preview | 39.5 | IBM Research ScarfBench post, October 27, 2026 (Gemini 3.1 Pro (OpenHands)) · IBM |