LongBench v2
General QA · 2024-12-19
LongBench v2 is a long-context benchmark of 503 four-option multiple-choice questions over contexts from 8K to 2M words, covering single- and multi-document QA, long in-context learning, long dialogue history, code repositories and long structured data. Scores are accuracy. Higher is better.
Top models (higher is better)
| Model | Score |
|---|---|
| Opus 4.8 | 69.1 |
| Gemini 3 Pro Preview | 68.2 |
| GPT-5.6 Sol | 67.1 |
| Qwen3.8-Max | 66.3 |
| Qwen3.7-Max (source unspecified) | 65.3 |
| Opus 4.5 | 64.4 |
| DeepSeek-V4-Pro | 62.1 |
| Qwen3.6 Plus (2026-04-02) | 62.0 |
| Nemotron 3 Ultra 550B A55B | 61.9 |
| Kimi K2.5 | 61.0 |
| GLM-5 | 60.8 |
| Qwen3 Max Thinking (2026-01-23) | 60.6 |
| Qwen3.5-27B | 60.6 |
| Qwen3.5 122B A10B | 60.2 |
| Qwen3.5 Plus (2026-02-15) | 60.2 |
| Qwen3.5-4B | 59.9 |
| DeepSeek-V3.2 | 59.8 |
| Qwen3.5-Omni-Plus | 59.6 |
| Qwen3.5 35B A3B | 59.0 |
| DeepSeek-V4-Flash | 57.0 |
| GPT-5 Mini | 56.8 |
| Qwen3.5-9B | 55.2 |
| Qwen3-235B-A22B | 54.8 |
| GPT-5.2 | 54.5 |
| GPT-4.1 | 54.3 |
| Sonnet 4 | 52.5 |
| DeepSeek-V4-Pro-Base | 51.5 |
| DeepSeek-V3-0324 | 51.1 |
| Nemotron 3.5 Bolt E4B BF16 | 50.5 |
| Gemma 4 E4B IT | 49.3 |
| Kimi K2 Instruct | 49.1 |
| DeepSeek-V3 | 48.7 |
| gpt-oss-120b | 48.2 |
| GPT-4o (2024-05-13) | 48.1 |
| Qwen3-Next-80B-A3B-Thinking | 48.0 |
| Qwen3.5-Omni-Flash | 46.4 |
| gpt-oss-20b | 45.6 |
| Qwen3 30B A3B Thinking 2507 | 44.8 |
| DeepSeek-V4-Flash-Base | 44.7 |
| Qwen3 4B Thinking 2507 | 42.8 |