← All benchmarks

VulcanBench Frontier v4

Code · 2026-09-09

VulcanBench Frontier v4 (formerly VulcanBench-SWE v4) is Morgan Linton's suite of 23 behavioural-reconstruction coding tasks. Each ships a retired compiled binary, a written specification that has drifted from the binary's behaviour and a naive replacement implementation, which the agent repairs until hidden tests captured from the binary pass. Every model runs each task once in a commercial coding agent (Claude Code, Codex, Cursor's agent CLI, Muse Code, DeepSeek Harness or Mirror CLI) at a stated reasoning effort, within a 3-hour task bound. A run's combined score is 100 × (0.50 partial-credit functional score + 0.085 lint and complexity + 0.085 security + 0.33 Code quality), where Code quality is judged for a human reader by a two-model panel from labs other than the graded model's, normally Muse Spark 1.3 and Grok 4.6, and the score is the mean over a column's judged tasks. This is the September 2026 weighting; earlier cards gave Code quality 20%. The board marks columns judged by a substitute panel as not strictly comparable with the others.

Benchmark source: Vulcanbench

Top models (higher is better)