← All benchmarks

PrivacyLens - adjusted leakage rate

Safety · 2024-08-29

PrivacyLens tests privacy-norm awareness of assistants acting for a user: each of 493 prompts gives the available tools and a pre-executed trajectory of tool calls, and the model writes one message, post or email. An LLM judge marks whether any of the scenario's sensitive items is exposed and rates helpfulness from 0 to 3. The adjusted leakage rate is the share of responses rated helpful (2 or above) that expose an item; lower is better.

Top models (lower is better)