RULER (4K-128K average)
General QA · 2024-04-09
RULER is NVIDIA's synthetic long-context benchmark of 13 configurable tasks spanning needle-in-a-haystack retrieval, multi-hop variable tracing, aggregation and question answering, generated at a chosen input length. This row is the suite's headline score, the average of the per-length scores at 4K, 8K, 16K, 32K, 64K and 128K tokens; every individual length, including the 256K-1M extensions, is its own child benchmark, and the family's primary is the better-covered 128K length (ruler-128k). Higher is better.
Top models (higher is better)
| Model | Score |
|---|---|
| Qwen2.5 14B Instruct 1M | 95.7 |
| Qwen2.5-Turbo | 93.1 |
| Qwen2.5 7B Instruct 1M | 91.8 |
| Qwen2.5 72B Instruct | 90.8 |
| Llama 3.1 70B Instruct | 89.6 |
| Qwen2.5 Instruct 32B | 88.0 |
| GPT-4o Mini | 87.3 |
| Qwen2.5 14B Instruct | 86.5 |
| Qwen2.5 7B Instruct | 80.1 |