BinaryAudit (False Positive Rate)
Cybersecurity · 2026-02-10
BinaryAudit's false positive rate: on eight control tasks with clean, unmodified binaries of lighttpd, dnsmasq, Dropbear and Sozu, the share of a model's runs that report a backdoor that is not there. Lower is better. The board counts between 18 and 24 of the 24 scheduled runs per model, and a model that rarely commits to a finding can score well, so it is a diagnostic rather than a capability score.
Top models (lower is better)
| Model | Score |
|---|---|
| GPT-5.2-Codex | 0.0 |
| GPT-5.2 | 0.0 |
| GPT-5.3-Codex | 0.0 |
| GPT-5.4 | 0.0 |
| GPT-5 | 0.0 |
| Grok 4.1 Fast | 0.0 |
| DeepSeek-V3.2 | 9.1 |
| Gemini 3.1 Pro Preview | 16.7 |
| Sonnet 4 | 18.2 |
| Opus 4.6 | 21.7 |
| GLM-4.7 | 21.7 |
| Opus 4.5 | 25.0 |
| Sonnet 4.5 | 27.3 |
| Gemini 3 Flash Preview | 31.6 |
| Haiku 4.5 | 33.3 |
| GLM-5 | 37.5 |
| Grok 4 | 47.8 |
| Kimi K2.5 | 54.2 |
| Gemini 3 Pro Preview | 65.2 |
| Gemini 2.5 Pro | 85.7 |