RULER - 32K
General QA · 2024-04-09
RULER score at a 32K-token input length: the average accuracy over RULER's 13 synthetic retrieval, multi-hop tracing, aggregation and question-answering tasks generated at that length. Scores at different lengths are kept as separate benchmarks.
Top models (higher is better)
| Model | Score |
|---|---|
| Qwen2.5 72B Instruct | 96.5 |
| Qwen2.5 Instruct 32B | 95.5 |
| Qwen2.5 14B Instruct 1M | 94.9 |
| Llama 3.1 70B Instruct | 94.8 |
| Qwen2.5-Turbo | 94.8 |
| Qwen2.5 14B Instruct | 93.4 |
| Qwen2.5 7B Instruct 1M | 91.1 |
| GPT-4o Mini | 90.2 |
| Qwen2.5 7B Instruct | 89.4 |