DROP
General QA · 2019-03-01
DROP (Discrete Reasoning Over Paragraphs) is a reading-comprehension benchmark whose questions need discrete operations such as addition, counting and sorting over Wikipedia passages. Scores are the token-level F1 (0-100) on the 9,536-question development set that model reports conventionally use; few-shot counts vary by source and are recorded per observation. Higher is better.
Top models (higher is better)
| Model | Score |
|---|---|
| DeepSeek-R1 | 92.2 |
| DeepSeek-V3 | 91.6 |
| o1 (2024-12-17) | 90.2 |
| o3 | 89.8 |
| DeepSeek-R1-Zero | 89.1 |
| DeepSeek-V3-Base | 89.0 |
| Llama 3.1 405B Instruct | 88.7 |
| DeepSeek-V4-Pro-Base | 88.7 |
| DeepSeek-V4-Flash-Base | 88.6 |
| Claude 3.5 Sonnet (Oct 2024) | 88.3 |
| DeepSeek-V2.5 | 87.8 |
| Claude 3.5 Sonnet (June 2024) | 87.1 |
| o1-mini | 83.9 |
| GPT-4.5 | 83.4 |
| Haiku 3.5 | 83.1 |
| Opus 3 | 83.1 |
| DeepSeek-V2-Chat | 83.0 |
| Gemini 1.0 Ultra | 82.4 |
| GPT-4.1 Nano | 82.2 |
| PaLM 2-L | 82.0 |
| GPT-4o (2024-11-20) | 81.5 |
| GPT-4.1 Mini | 81.0 |
| GPT-4 (source unspecified) | 80.9 |
| Qwen2.5 72B | 80.6 |
| o3-mini | 80.6 |
| GPT-4o (2024-08-06) | 79.8 |
| GPT-4o Mini | 79.7 |
| Llama 3.1 70B | 79.6 |
| GPT-4.1 | 79.4 |
| Claude 3 Sonnet | 78.9 |
| Gemini 1.5 Pro (February 2024 report) | 78.9 |
| Claude 3 Haiku | 78.4 |
| Gemini 1.5 Flash (May 2024 report) | 78.4 |
| O4 Mini | 78.1 |
| Gemma 3 27B PT | 77.2 |
| Qwen2.5 72B Instruct | 76.7 |
| Gemini 1.5 Pro (May 2024) | 74.9 |
| o1 Preview | 74.8 |
| Gemini 1.0 Pro | 74.1 |
| Gemma 3 12B PT | 72.2 |