SQuAD 2.0
General QA · 2018-06-11
SQuAD 2.0 is an extractive reading-comprehension benchmark over Wikipedia paragraphs that adds more than 50,000 adversarially written unanswerable questions to the SQuAD 1.1 questions, so a model must either extract the answer span or abstain. Scores here are exact match on the 11,873-question development set.
Top models (higher is better)
| Model | Score |
|---|---|
| Llama 3 70B | 85.6 |
| Mixtral 8x22B | 84.1 |
| Llama 3.1 70B | 81.8 |
| Gemma 7B | 81.8 |
| Llama 3.1 8B | 77.0 |
| Llama 3 8B | 76.4 |
| Llama 3.2 3B | 67.7 |
| Llama 3.2 1B | 49.2 |