Benchmarks
347 sources · 1,258 models × 3,038 benchmarks · 158,912 scoresCyberBinGym tests 1-day binary exploitation: an agent gets only the stripped, sanitizer-free vulnerable and patched builds of a real open-source program, with Ghidra, Radare2 and GDB and network access to a victim service, but no source code, source patch, vulnerability description or crashing input. The score is the share of 100 tasks (53 projects, built from CyberGym's ARVO and OSS-Fuzz vulnerabilities) on which the agent retrieves the victim's secret flag through vulnerability-driven behaviour, confirmed by a manual trajectory audit, in one trajectory per task. Unlike ExploitGym, which supplies source, a description and a triggering input, the agent must first localize the bug from the binary patch. Higher is better.
No signal or difficulty estimate. Too few models have been measured on this benchmark to estimate its difficulty.
| # | Model | Setting | Score | Source |
|---|---|---|---|---|
| 1 | GPT-5.6 Sol | not reported · CyberBinGym Inspect sandbox (static and dynamic analysis containers, a victim service, proxy-restricted egress); Codex CLI with web search disabled; nominal 1,000 LLM-call budget; trusted-access program with safeguards disabled | 20.0 | |
| 2 | Claude Opus 5 | not reported · CyberBinGym Inspect sandbox (static and dynamic analysis containers, a victim service, proxy-restricted egress); Claude Code with its Bash and WebSearch tools replaced by a bridged shell; nominal 1,000 LLM-call budget; trusted-access program with safeguards disabled | 6.0 | |
| 3 | Kimi K3 | not reported · CyberBinGym Inspect sandbox (static and dynamic analysis containers, a victim service, proxy-restricted egress); Kimi Code with WebSearch and FetchURL disabled; nominal 1,000 LLM-call budget; trusted-access program with safeguards disabled | 0.0 |