RULER - 16K
General QA · 2024-04-09
RULER score at a 16K-token input length: the average accuracy over RULER's 13 synthetic retrieval, multi-hop tracing, aggregation and question-answering tasks generated at that length. Scores at different lengths are kept as separate benchmarks.
Top models (higher is better)
| Model | Score |
|---|---|
| Qwen2.5 72B Instruct | 97.7 |
| Qwen2.5 14B Instruct | 95.9 |
| Qwen2.5 Instruct 32B | 95.5 |
| Qwen2.5-Turbo | 95.5 |
| Llama 3.1 70B Instruct | 95.4 |
| Qwen2.5 14B Instruct 1M | 94.6 |
| Qwen2.5 7B Instruct | 93.7 |
| Qwen2.5 7B Instruct 1M | 93.0 |
| GPT-4o Mini | 92.7 |