BinaryAudit (Binary Analysis)
Cybersecurity · 2026-02-10
Quesma's BinaryAudit asks an agent with Ghidra, Radare2 and binutils to decide whether stripped binaries of real network software (lighttpd, dnsmasq, Dropbear, Sozu) contain backdoors or timebombs that Quesma injected, without source code. This Binary Analysis score covers the 19 detection tasks: a run passes when it identifies the malicious code as the task asks, for example the address of the function holding the backdoor, which of several binaries are altered, or a timebomb's trigger. Each model runs three times per task in Harbor, and the score is the share of its 57 runs that pass.
Top models (higher is better)
| Model | Score |
|---|---|
| Opus 4.6 | 49.1 |
| Gemini 3.1 Pro Preview | 49.1 |
| GPT-5.2-Codex | 45.6 |
| Gemini 3 Pro Preview | 43.9 |
| GPT-5.3-Codex | 42.1 |
| Opus 4.5 | 36.8 |
| Gemini 3 Flash Preview | 36.8 |
| Sonnet 4.5 | 26.3 |
| GLM-4.7 | 24.6 |
| Kimi K2.5 | 24.6 |
| Gemini 2.5 Pro | 21.1 |
| GLM-5 | 19.3 |
| GPT-5.2 | 17.5 |
| GPT-5.4 | 17.5 |
| Sonnet 4 | 14.0 |
| DeepSeek-V3.2 | 12.3 |
| Haiku 4.5 | 10.5 |
| Grok 4 | 8.8 |
| Grok 4.1 Fast | 7.0 |
| GPT-5 | 1.8 |