Atlas

Benchmarks

CyberBinGym

CyberBinGym tests 1-day binary exploitation: an agent gets only the stripped, sanitizer-free vulnerable and patched builds of a real open-source program, with Ghidra, Radare2 and GDB and network access to a victim service, but no source code, source patch, vulnerability description or crashing input. The score is the share of 100 tasks (53 projects, built from CyberGym's ARVO and OSS-Fuzz vulnerabilities) on which the agent retrieves the victim's secret flag through vulnerability-driven behaviour, confirmed by a manual trajectory audit, in one trajectory per task. Unlike ExploitGym, which supplies source, a description and a triggering input, the agent must first localize the bug from the binary patch. Higher is better.

Models
3
Top score
20.0
Leader
GPT-5.6 Sol
Updated
Nov 7, 2026
Resolution
—
Saturation
—

No signal or difficulty estimate. Too few models have been measured on this benchmark to estimate its difficulty.

Organizations
3 dated models · 1 frontier modelsClosed weightsOpen weights
Data: CyberBinGym Authorsatlas.village.dev

CyberBinGym leaderboard

#ModelSettingScoreSource
1GPT-5.6 Solnot reported · CyberBinGym Inspect sandbox (static and dynamic analysis containers, a victim service, proxy-restricted egress); Codex CLI with web search disabled; nominal 1,000 LLM-call budget; trusted-access program with safeguards disabled
20.0
2Claude Opus 5not reported · CyberBinGym Inspect sandbox (static and dynamic analysis containers, a victim service, proxy-restricted egress); Claude Code with its Bash and WebSearch tools replaced by a bridged shell; nominal 1,000 LLM-call budget; trusted-access program with safeguards disabled
6.0
3Kimi K3not reported · CyberBinGym Inspect sandbox (static and dynamic analysis containers, a victim service, proxy-restricted egress); Kimi Code with WebSearch and FetchURL disabled; nominal 1,000 LLM-call budget; trusted-access program with safeguards disabled
0.0
Loading Atlas data…