Benchmarks
347 sources · 1,258 models × 3,038 benchmarks · 158,912 scoresHanno Hilbig's (UC Davis) benchmark of language models as coders of political-science text: 33 classification tasks drawn from published papers, replication archives and public datasets (relevance, incivility, stance, sentiment, events, protest forms and claims, causal relations, policy topics), with 100 texts per task sampled under a fixed seed. Every model codes every text once with the same prompt and structured output, and is scored against the original authors' human labels: positive-class F1 for binary tasks, mean positive-class F1 for the one multi-label task and macro F1 over the classes present in the sample for categorical tasks, with unusable answers counted wrong. The score is the equal-weight mean over the 33 tasks. Reasoning is turned off where a model allows it and otherwise set to its lowest level.
697th strongest of 798 fitted benchmarks
Signal-to-noise +2.58
790th hardest of 798 fitted benchmarks
Capability needed ≲ 63.3
| # | Model | Setting | Score | Source |
|---|---|---|---|---|
| 1 | Claude Opus 5 | disabled | 71.4 | |
| 2 | Claude Opus 5.5 | low | 70.2 | |
| 3 | Sonnet 5 | disabled | 69.7 | |
| 4 | Gemini 4 Argon | low | 69.6 | |
| 5 | GPT-5.6 Sol | disabled | 69.2 | |
| 6 | Claude Sonnet 5.5 | low | 68.9 | |
| 7 | Gemini 3.8 Flash | disabled | 68.7 | |
| 8 | Kimi K3.1 Pro | disabled | 68.6 | |
| 9 | GPT-6.1 Sol | low | 68.3 | |
| 10 | GPT-5.6 Terra | disabled | 68.3 | |
| 11 | Gemini 3.1 Flash-Lite | disabled | 68.1 | |
| 12 | GPT-6 Astra | low | 67.9 | |
| 13 | GPT-5.6 Luna | disabled | 67.0 | |
| 14 | GLM-5.3 | low | 66.7 | |
| 15 | GPT-6 Luna | disabled | 66.6 | |
| 16 | Grok 4.7 | low | 66.6 | |
| 17 | DeepSeek V4.1 Flash | disabled | 66.5 | |
| 18 | DeepSeek V4 Pro 0813 | disabled | 66.3 | |
| 19 | DeepSeek V4.1 Flash 1017 | disabled | 66.2 | |
| 20 | Beam 1.0 501B A23B | low | 66.2 | |
| 21 | Claude Haiku 5.5 | disabled | 65.7 | |
| 22 | Qwen2.5 Instruct 32B | not supported | 64.9 | |
| 23 | Qwen3 32B | disabled | 64.8 | |
| 24 | Mistral Small 3.1 24B Instruct 2503 | not supported | 64.2 | |
| 25 | DeepSeek-R1-Distill-Qwen-32B | always on | 63.4 | |
| 26 | Qwen3 30B A3B Instruct 2507 FP8 | not supported | 63.0 | |
| 27 | Qwen1.5 32B Chat | not supported | 61.9 | |
| 28 | GLM 4.7 Flash | disabled | 60.2 | |
| 29 | Qwen3-30B-A3B | disabled | 59.3 |
Benchmarks outside its family whose model rankings track it most reliably (Spearman ρ over at least 10 shared models).