RULER - 8K
General QA · 2024-04-09
RULER score at a 8K-token input length: the average accuracy over RULER's 13 synthetic retrieval, multi-hop tracing, aggregation and question-answering tasks generated at that length. Scores at different lengths are kept as separate benchmarks.
Top models (higher is better)
| Model | Score |
|---|---|
| Qwen2.5 72B Instruct | 97.2 |
| Qwen2.5 Instruct 32B | 97.1 |
| Qwen2.5 14B Instruct 1M | 97.1 |
| Qwen2.5 14B Instruct | 96.8 |
| Llama 3.1 70B Instruct | 95.8 |
| Qwen2.5-Turbo | 95.7 |
| Qwen2.5 7B Instruct 1M | 95.3 |
| Qwen2.5 7B Instruct | 95.1 |
| GPT-4o Mini | 92.9 |