GPQA Main

Name: GPQA Main
Published: 2023-11-20

General QA · 2023-11-20

GPQA Main is the 448-question main set of GPQA, graduate-level four-option multiple-choice questions in biology, physics and chemistry written by domain experts to be hard to answer with web search. It is broader and easier than the 198-question Diamond subset, so the two are kept apart. Scores are accuracy.

Top models (higher is better)

Model	Score
Llama 3.3 70B Instruct	53.1
Llama 3.1 70B Instruct	46.7
Llama 3.2 90B Vision Instruct	46.7
Mistral Small 3 24B Instruct 2501	45.3
Mistral Small 3.1 24B Instruct 2503	44.4
Mistral Small 3.2 24B Instruct 2506	44.2
Qwen2.5 Instruct 32B	40.4
Llama 3 70B Instruct	39.5
Mistral Small 3.1 24B Base 2503	37.5
Haiku 3.5	37.0
Gemma 3 27B IT	36.8
Gemma 2 27B IT	34.4
Mistral Small 3 24B Base 2501	34.4
Llama 3.2 3B Instruct	32.8
Llama 3.2 11B Vision Instruct	32.8
Phi-3.5 Mini Instruct	31.9
Gemma 2 2B IT	27.5
Llama 3.2 1B Instruct	27.2
Llama 2 13B Chat	22.3
Llama 2 7B Chat	21.7
Llama 2 70B Chat	21.0