CIMemories - coverage rate
Safety · 2025-11-18
CIMemories pairs synthetic user profiles of over 100 personal attributes with task contexts, giving 2,500 scenarios in which each attribute is either necessary or inappropriate to share. The coverage rate is the extent to which necessary attributes are shared, as judged by an LLM, and is the task-completion counterpart to the violation rate.
Top models (higher is better)
| Model | Score |
|---|---|
| Qwen3.6 27B | 66.9 |
| Muse Glimmer 30B | 66.4 |
| Gemma 4 31B IT | 53.0 |