Social IQa (OLMo 3 5-shot multiple choice)
General QA · 2025-12-15
Social IQa as Ai2 runs it in the OLMo 3 base-model evaluation (OlmoBaseEval "SocialIQA MC"): five-shot, with the three options lettered and the model choosing a letter, scored as accuracy. Asked this way, strong base models reach the 80s, while likelihood scoring of the same questions plateaus near 50%, so this protocol is its own factor group. Ai2's suite draws its instances from the full dataset rather than the 1,954-question development split, so the item count is not stated.
Top models (higher is better)
| Model | Score |
|---|---|
| Qwen2.5 32B | 86.6 |
| Llama 3.1 70B | 83.5 |
| Qwen3 8B Base | 83.3 |
| Qwen2.5 7B | 82.9 |
| OLMo 2 32B | 82.3 |
| Gemma 3 27B PT | 81.2 |
| Mistral Small 3.1 24B Base 2503 | 81.0 |
| Gemma 2 27B | 81.0 |
| Gemma 2 9B | 80.3 |
| OLMo 2 7B | 77.5 |
| Llama 3.1 8B | 77.0 |