Natural Questions (open-domain, closed-book)
General QA · 2019-01-24
NaturalQuestions-Open: real anonymized Google Search questions answered without retrieved context (closed-book), scored by exact match against the annotated short answers. The canonical split is the 3,610-question NQ-Open test set; a few reports use the development split, and shot counts (commonly 5) are recorded per observation. Higher is better.
Top models (higher is better)
| Model | Score |
|---|---|
| Mixtral 8x22B Instruct | 54.9 |
| DeepSeek-V2-Chat | 53.4 |
| DeepSeek-Coder-V2-Instruct | 47.5 |
| DeepSeek LLM 67B Chat | 47.0 |
| Qwen1.5 72B Chat | 46.9 |
| Llama 3 70B Instruct | 44.6 |
| Llama 3.1 405B | 41.5 |
| Llama 3 70B | 40.2 |
| DeepSeek-V3-Base | 40.0 |
| PaLM 2-L | 37.5 |
| DeepSeek LLM 67B Base | 36.6 |
| Gemma 3 27B PT | 36.1 |
| Qwen1.5 72B | 35.6 |
| DeepSeek-V2-Lite-Chat | 35.5 |
| DeepSeekMoE 16B Chat | 35.1 |
| Mistral Small 3.1 24B Base 2503 | 34.4 |
| Qwen2.5 72B | 33.2 |
| PaLM 2-M | 32.0 |
| Gemma 3 12B PT | 31.4 |
| Mistral NeMo Base 2407 | 31.2 |
| DeepSeek-Coder-V2-Lite-Instruct | 30.8 |
| Ministral 3 14B Base 2512 | 29.9 |
| Llama 3 8B | 28.2 |
| DeepSeek-V2-Lite | 26.0 |
| Ministral 3 8B Base 2512 | 25.8 |
| DeepSeekMoE 16B Base | 25.5 |
| PaLM 2-S | 25.3 |
| Gemini Nano 2 | 24.8 |
| Gemma 7B | 23.0 |
| DeepSeek LLM 7B Base | 22.2 |
| Ministral 3 3B Base 2512 | 21.9 |
| RecurrentGemma 9B | 21.7 |
| Gemma 3n E4B | 20.9 |
| Gemma 3 4B PT | 20.0 |
| Gemini Nano 1 | 18.8 |
| Gemma 3n E2B | 15.5 |
| Gemma 2B | 12.5 |
| RecurrentGemma 2B | 11.5 |
| Gemma 3 1B PT | 9.5 |