← All benchmarks

ASPI - clarification-state ask_user attack success rate

Safety · 2026-05-17

In ASPI's clarification conditions each of the 728 AgentDojo task-attack groups gets a prompt that omits one required user-provided slot, and a GPT-4o-mini user simulator supplies the missing information when the agent calls the ask_user tool (an agent that asks in plain text gets one retry with an explicit instruction, then the group is dropped). In this condition (clarif_ask_user) the attacker's instruction is embedded in that ask_user return value. The attack success rate is computed only over the groups in which the model invoked ask_user (its clarification rate times 728), so the denominator differs by model; lower is better, and it measures robustness to prompt injection, not capability.

Top models (lower is better)