BinaryAudit (Tooling)
Cybersecurity · 2026-02-10
BinaryAudit's six tooling tasks check that an agent can operate the reverse-engineering tools the benchmark provides, decompiling a binary with Ghidra (directly or through PyGhidra, with and without jq) and with Radare2. The score is the share of a model's 18 runs (three per task) that pass. It is a component of the BinaryAudit factor group.
Top models (higher is better)
| Model | Score |
|---|---|
| Opus 4.5 | 100.0 |
| Opus 4.6 | 100.0 |
| Gemini 3.1 Pro Preview | 100.0 |
| Gemini 3 Pro Preview | 100.0 |
| GLM-5 | 100.0 |
| GPT-5.2-Codex | 100.0 |
| GPT-5.3-Codex | 100.0 |
| Sonnet 4.5 | 94.4 |
| GPT-5.4 | 94.4 |
| GPT-5.2 | 88.9 |
| Kimi K2.5 | 88.9 |
| GPT-5 | 83.3 |
| GLM-4.7 | 66.7 |
| Haiku 4.5 | 61.1 |
| DeepSeek-V3.2 | 61.1 |
| Gemini 3 Flash Preview | 61.1 |
| Grok 4 | 61.1 |
| Sonnet 4 | 50.0 |
| Grok 4.1 Fast | 50.0 |
| Gemini 2.5 Pro | 22.2 |