Siren AgentDojo - adaptive attack success rate
Safety · 2024-06-19
Siren AgentDojo runs AgentDojo's 97 benign tool-use tasks in banking, travel, workspace and online-store environments against 35 injected malicious goals, 949 scenarios in all, with an LLM attacker that refines its injection over several attempts. The attack success rate is the share of scenarios in which the agent carries out the injected malicious task; lower is better.
Top models (lower is better)
| Model | Score |
|---|---|
| Gemma 4 31B IT | 25.6 |
| Muse Glimmer 30B | 27.8 |
| Qwen3.6 27B | 40.3 |