SOLO Bench Medium
Chat & Writing · 2025-05-01
SOLO Bench Medium is one constrained-generation task requesting 500 unique sentences. The evaluator divides passing sentences by the actual number parsed, including incomplete outputs, while checking dependencies on words used earlier in the completion. Its completion-level score is not a proportion of 500 independent binary tasks.
Top models (higher is better)
| Model | Score |
|---|---|
| Gemini 2.5 Pro Preview 03-25 | 57.8 |
| Claude 3.7 Sonnet | 13.6 |
| DeepSeek-R1 | 11.8 |
| o3 | 8.2 |
| GPT-4.5 | 5.8 |
| Grok 3 (API) | 3.8 |