Benchmarks
347 sources · 1,258 models × 3,038 benchmarks · 158,912 scoresThe share of CyberBinGym's 100 binary-only tasks on which the agent's submitted input triggers the expected sanitizer-detected failure in a hidden instrumented build of the vulnerable version while the patched build stays clean: CyberGym's differential proof-of-concept test, here from stripped binaries alone. It is scored independently of exploitation, since an accepted exploit may use a different bug and need not pass this test. Higher is better.
No signal or difficulty estimate. Too few models have been measured on this benchmark to estimate its difficulty.
| # | Model | Setting | Score | Source |
|---|---|---|---|---|
| 1 | GPT-5.6 Sol | not reported · CyberBinGym Inspect sandbox (static and dynamic analysis containers, a victim service, proxy-restricted egress); Codex CLI with web search disabled; nominal 1,000 LLM-call budget; trusted-access program with safeguards disabled | 88.0 | |
| 2 | Claude Opus 5 | not reported · CyberBinGym Inspect sandbox (static and dynamic analysis containers, a victim service, proxy-restricted egress); Claude Code with its Bash and WebSearch tools replaced by a bridged shell; nominal 1,000 LLM-call budget; trusted-access program with safeguards disabled | 85.0 | |
| 3 | Kimi K3 | not reported · CyberBinGym Inspect sandbox (static and dynamic analysis containers, a victim service, proxy-restricted egress); Kimi Code with WebSearch and FetchURL disabled; nominal 1,000 LLM-call budget; trusted-access program with safeguards disabled | 64.0 |