COLLIE
Chat & Writing · 2023-07-17
COLLIE (Constrained Language Generation) tests whether a model can write free-form text that satisfies compositional, grammar-defined constraints at word, sentence and paragraph level, using the COLLIE-v1 instance set. A response scores only if every constraint is met; the reported score is the share of satisfied instances. Higher is better.
Top models (higher is better)
| Model | Score |
|---|---|
| GPT-5 | 99.0 |
| o3-mini | 98.7 |
| GPT-5 Mini | 98.5 |
| o3 | 98.4 |
| GPT-5 Nano | 96.9 |
| O4 Mini | 96.1 |
| o1 (2024-12-17) | 95.3 |
| GPT-4.5 | 72.3 |
| GPT-4.1 | 65.8 |
| GPT-4.1 Mini | 54.6 |
| GPT-4o Mini | 52.7 |
| GPT-4o (2024-11-20) | 50.2 |
| GPT-4.1 Nano | 42.5 |