RULER - 4K
General QA · 2024-04-09
RULER score at a 4K-token input length: the average accuracy over RULER's 13 synthetic retrieval, multi-hop tracing, aggregation and question-answering tasks generated at that length. Scores at different lengths are kept as separate benchmarks.
Top models (higher is better)
| Model | Score |
|---|---|
| Qwen2.5 14B Instruct | 97.7 |
| Qwen2.5 72B Instruct | 97.7 |
| Qwen2.5 14B Instruct 1M | 97.5 |
| Qwen2.5-Turbo | 97.5 |
| Qwen2.5 Instruct 32B | 96.9 |
| Qwen2.5 7B Instruct 1M | 96.8 |
| Qwen2.5 7B Instruct | 96.7 |
| Llama 3.1 70B Instruct | 96.5 |
| GPT-4o Mini | 95.0 |