← All benchmarks

CWE-bench v2 Pass@1

Cybersecurity · 2026-11-02

Collinear AI's 250-task defensive security audit-and-patch evaluation across 185 CWEs and six languages: 100 public tasks and 150 held-out tasks. Models receive four high-reasoning rollouts per task, each limited to one hour; a rollout that reaches the limit is graded on the code at that point, and one lost to infrastructure is rerun. A deterministic verifier requires the exploit to fail and legitimate behaviour to hold, with no partial credit. Pass@1 averages successful rollouts per task. This is a new task cohort, not a rescoring of the 120-task CWE-bench v1.

Benchmark source: Cwe Bench

Top models (higher is better)