GPQA (Open LLM Leaderboard v2)

Name: GPQA (Open LLM Leaderboard v2)
Published: 2024-06-26

General QA · 2024-06-26

The Open LLM Leaderboard v2 GPQA figure is a size-weighted mean of the extended (546), main (448), and diamond (198) split accuracies. Main and diamond are subsets of extended, so there are 546 distinct questions and unequal repeated-question weights, not 1,192 independent items. It uses the leaderboard's fixed lm-evaluation-harness prompt without chain-of-thought and records raw accuracy rather than the baseline-normalized column; this exact aggregation has its own factor group.

Top models (higher is better)

Model	Score
Mistral Large 2.1 (Instruct 2411)	43.7
Qwen2.5 32B	41.2
Phi-4	40.6
Qwen2.5 72B	40.5
Qwen2-72B	39.4
DeepSeek R1 Distill Qwen 14B	38.8
Llama 3.1 70B	38.8
Qwen2 VL 72B Instruct	38.8
Llama 3.1 Tülu 3 70B DPO	37.6
Mixtral 8x22B	37.6
Gemma 2 27B IT	37.5
Mixtral 8x22B Instruct	37.3
Qwen2 72B Instruct	37.2
Solar Pro Instruct Preview	37.1
Yi-34B	36.7
Yi 1.5 34B Chat	36.5
Aria	36.2
Hermes 3 Llama 3.1 70B	36.2
Gemma 2 9B IT	36.1
Llama 3.1 70B Instruct	35.7
Phi-3.5-MoE-instruct	35.6
C4AI Command R+ 08-2024	35.1
Gemma 2 27B	35.1
Qwen2.5 Coder 32B Instruct	34.9
DBRX Instruct	34.1
Qwen1.5 110B Chat	34.1
Phi-3.5 Mini Instruct	34.0
Qwen2.5 Instruct 32B	33.8
Yi 34B Chat	33.8
Phi-3 Medium 128K Instruct	33.6
Phi-3 Medium 4K Instruct	33.6
Yi 1.5 9B Chat	33.5
Qwen1.5 32B	33.0
Gemma 2 9B	32.9
Llama 3.3 70B Instruct	32.9
Aya Expanse 32B	32.6
Mistral Small Instruct 2409	32.4
Smaug-72B-v0.1	32.4
Qwen2.5 14B Instruct	32.2
Phi-3-mini-4k-instruct	32.0