Atlas

Benchmarks

Political Science LLM Benchmark

Hanno Hilbig's (UC Davis) benchmark of language models as coders of political-science text: 33 classification tasks drawn from published papers, replication archives and public datasets (relevance, incivility, stance, sentiment, events, protest forms and claims, causal relations, policy topics), with 100 texts per task sampled under a fixed seed. Every model codes every text once with the same prompt and structured output, and is scored against the original authors' human labels: positive-class F1 for binary tasks, mean positive-class F1 for the one multi-label task and macro F1 over the classes present in the sample for categorical tasks, with unusable answers counted wrong. The score is the equal-weight mean over the 33 tasks. Reasoning is turned off where a model allows it and otherwise set to its lowest level.

Models
29
Top score
71.4
Leader
Claude Opus 5
Updated
—
Resolution
~8 mo
Coarse
Saturation
after 2031
Projected
Signal
13thpercentile
195 benchmarks from 0.8 to 3.5224 benchmarks from 3.5 to 6.1133 benchmarks from 6.1 to 8.898 benchmarks from 8.8 to 11.566 benchmarks from 11.5 to 14.126 benchmarks from 14.1 to 16.815 benchmarks from 16.8 to 19.59 benchmarks from 19.5 to 22.18 benchmarks from 22.1 to 24.84 benchmarks from 24.8 to 27.55 benchmarks from 27.5 to 30.12 benchmarks from 30.1 to 32.84 benchmarks from 32.8 to 35.45 benchmarks from 35.4 to 38.11 benchmark from 38.1 to 40.82 benchmarks from 43.4 to 46.11 benchmark from 62.1 to 64.8

697th strongest of 798

Signal-to-noise +2.58

Difficulty
1stpercentile
1 benchmark from 5.5 to 14.41 benchmark from 14.4 to 23.22 benchmarks from 32.1 to 41.01 benchmark from 41.0 to 49.82 benchmarks from 49.8 to 58.79 benchmarks from 58.7 to 67.67 benchmarks from 67.6 to 76.418 benchmarks from 76.4 to 85.324 benchmarks from 85.3 to 94.241 benchmarks from 94.2 to 103.031 benchmarks from 103.0 to 111.939 benchmarks from 111.9 to 120.850 benchmarks from 120.8 to 129.676 benchmarks from 129.6 to 138.560 benchmarks from 138.5 to 147.4118 benchmarks from 147.4 to 156.2124 benchmarks from 156.2 to 165.1120 benchmarks from 165.1 to 174.039 benchmarks from 174.0 to 182.820 benchmarks from 182.8 to 191.710 benchmarks from 191.7 to 200.61 benchmark from 200.6 to 209.44 benchmarks from 209.4 to 218.3

790th hardest of 798

Capability needed ≲ 63.3

Organizations
28 dated models · 5 frontier modelsClosed weightsOpen weights
Data: University of California, Davisatlas.village.dev

Political Science LLM Benchmark leaderboard

#ModelSettingScoreSource
1Claude Opus 5disabled
71.4
2Claude Opus 5.5low
70.2
3Sonnet 5disabled
69.7
4Gemini 4 Argonlow
69.6
5GPT-5.6 Soldisabled
69.2
6Claude Sonnet 5.5low
68.9
7Gemini 3.8 Flashdisabled
68.7
8Kimi K3.1 Prodisabled
68.6
9GPT-6.1 Sollow
68.3
10GPT-5.6 Terradisabled
68.3
11Gemini 3.1 Flash-Litedisabled
68.1
12GPT-6 Astralow
67.9
13GPT-5.6 Lunadisabled
67.0
14GLM-5.3low
66.7
15GPT-6 Lunadisabled
66.6
16Grok 4.7low
66.6
17DeepSeek V4.1 Flashdisabled
66.5
18DeepSeek V4 Pro 0813disabled
66.3
19DeepSeek V4.1 Flash 1017disabled
66.2
20Beam 1.0 501B A23Blow
66.2
21Claude Haiku 5.5disabled
65.7
22Qwen2.5 Instruct 32Bnot supported
64.9
23Qwen3 32Bdisabled
64.8
24Mistral Small 3.1 24B Instruct 2503not supported
64.2
25DeepSeek-R1-Distill-Qwen-32Balways on
63.4
26Qwen3 30B A3B Instruct 2507 FP8not supported
63.0
27Qwen1.5 32B Chatnot supported
61.9
28GLM 4.7 Flashdisabled
60.2
29Qwen3-30B-A3Bdisabled
59.3

Related benchmarks

Loading Atlas data…