Atlas

Benchmarks

← All benchmarks

GPQA - release-reported, subset unspecified

Science · 2023-11-20

GPQA graduate-level science multiple-choice results for which the reporting source names only "GPQA" and does not state whether the main, extended or Diamond question set was used. Retained as a diagnostic rather than pooled with GPQA Diamond.

Top models (higher is better)

ModelScore
o1 Preview72.3
QwQ-32B-Preview65.2
o1-mini60.0
Claude 3.5 Sonnet (June 2024)59.4
Mistral Large 2 (Instruct 2407)52.0
Llama 3.1 405B Instruct51.1
Qwen Plus (rolling alias)49.7
Mixtral 8x22B Instruct49.7
Qwen2.5 Instruct 32B49.5
Qwen2.5 72B Instruct49.0
Qwen2.5-VL-72B-Instruct49.0
Qwen2.5 32B48.0
Llama 3.1 70B Instruct46.7
Qwen2.5 72B45.9
Qwen2.5 14B Instruct45.5
DeepSeek-V2.545.5
Qwen-Turbo (rolling alias)44.4
Qwen2 72B Instruct42.4
Qwen2.5-Turbo42.3
Llama 3 70B Instruct41.9
Qwen2.5 Coder 32B Instruct41.8
Qwen2.5 7B Instruct 1M41.4
GPT-4o Mini40.2
Qwen2.5 14B Instruct 1M39.9
Qwen1.5 72B Chat39.4
Gemma 2 27B IT38.4
Qwen2.5 Coder 14B Instruct36.8
Qwen2.5 7B Instruct36.4
Qwen2.5 7B36.4
Qwen1.5 72B36.3
Qwen1.5 110B35.9
Qwen2.5-Coder-7B-Instruct35.6
Gemma 2 27B34.9
Mixtral 8x22B34.3
Qwen2-57B-A14B34.3
Qwen2-57B-A14B Instruct34.3
Llama 3 8B Instruct34.2
Qwen2.5 14B32.8
Gemma 2 9B32.8
Gemma 2 9B IT32.8
Loading Atlas data…