CIMemories - violation rate
Safety · 2025-11-18
CIMemories measures whether models with persistent memory reveal personal attributes in inappropriate contexts. Synthetic user profiles with over 100 attributes each are paired with task contexts such as writing to a doctor, giving 2,500 scenarios in which each attribute is either necessary or inappropriate. The violation rate is the extent to which inappropriate attributes are revealed, as judged by an LLM; lower is better.
Top models (lower is better)
| Model | Score |
|---|---|
| Gemma 4 31B IT | 12.1 |
| Muse Glimmer 30B | 25.5 |
| Qwen3.6 27B | 53.4 |