← All benchmarks

SWE-Clar (clarification)

Code · 2026-10-22

SWE-Clar's clarification score: from the same frozen clarification context as the implementation run, but in a separate isolated run, the agent answers the benchmark's 512 requirement questions about what the client needs. A judge grades each answer against reference answers from the real client-freelancer interactions (correct 1, partial 0.5, incorrect 0); answers are averaged within each task and then across the 100 tasks. It measures how well the agent elicited and understood the requirements rather than whether it built them, so it is a diagnostic beside the implementation score.

Top models (higher is better)