OpenBookQA
Science · 2018-09-08
OpenBookQA is a multiple-choice elementary science question-answering benchmark requiring use of a small open-book fact set plus common knowledge. It is often used to measure scientific commonsense reasoning.
Top models (higher is better)
| Model | Score |
|---|---|
| DeepSeek LLM 67B Chat | 63.2 |
| Mistral NeMo Base 2407 | 60.6 |
| DeepSeek LLM 67B Base | 60.2 |
| LLaMA 65B | 60.2 |
| Llama 2 70B | 60.2 |
| Llama 2 7B | 58.6 |
| LLaMA 33B | 58.6 |
| PaLM 2-L | 58.5 |
| Llama 2 34B | 58.2 |
| PaLM 2-S | 57.4 |
| LLaMA 7B | 57.2 |
| Llama 2 13B | 57.0 |
| Falcon 40B | 56.6 |
| LLaMA-13B | 56.4 |
| Llama 3 8B | 56.4 |
| PaLM 2-M | 56.2 |
| DeepSeek LLM 7B Base | 55.8 |
| DeepSeek LLM 7B Chat | 54.8 |
| Gemma 7B | 52.8 |
| MPT-30B | 52.0 |
| RecurrentGemma 9B | 51.8 |
| Falcon 7B | 51.6 |
| MPT-7B | 51.4 |
| Mixtral 8x22B | 50.8 |
| Gemma 2 9B | 50.8 |
| Kimi K2 Base | 50.8 |
| Llama 3.1 405B | 49.2 |
| Nemotron-H 47B Base 8K | 48.8 |
| Nemotron 3 Ultra 550B A55B Base | 48.6 |
| Nemotron-H 56B Base 8K | 48.6 |
| Gemma 2B | 47.8 |
| Llama 3.1 70B | 47.6 |
| Nemotron-H 8B Base 8K | 47.2 |
| RecurrentGemma 2B | 47.2 |
| Qwen2.5 72B | 46.2 |
| Nemotron Nano 12B V2 Base | 46.0 |
| Nemotron Nano 9B V2 Base | 44.8 |
| Nemotron-H 4B Base 8K | 44.2 |