Siren AgentDojo - utility under attack
Safety · 2024-06-19
Siren AgentDojo runs AgentDojo's 97 benign tool-use tasks in banking, travel, workspace and online-store environments against 35 injected malicious goals, 949 scenarios in all, with an LLM attacker that refines its injection over several attempts. Utility is the share of scenarios in which the agent still completes the user's benign task in the presence of the injection.
Top models (higher is better)
| Model | Score |
|---|---|
| Muse Glimmer 30B | 94.2 |
| Qwen3.6 27B | 92.7 |
| Gemma 4 31B IT | 90.8 |