BIG-Bench Extra Hard
General QA · 2025-02-26
BIG-Bench Extra Hard replaces each BIG-Bench Hard task with a more difficult task probing a similar reasoning ability. Scores report aggregate answer accuracy.
Top models (higher is better)
| Model | Score |
|---|---|
| Gemma 4 31B IT | 74.4 |
| Gemma 4 26B A4B IT | 64.8 |
| DiffusionGemma 26B A4B IT | 59.1 |
| Gemma 4 12B IT | 53.0 |
| Gemma 4 E4B IT | 33.1 |
| DeepSeek-V4-Pro-Base | 29.8 |
| DeepSeek-V4-Flash-Base | 25.4 |
| Gemma 4 E2B IT | 21.9 |
| Gemma 3 27B IT | 19.3 |
| Gemma 3 12B IT | 16.3 |
| Gemma 2 27B IT | 14.8 |
| Gemma 3 4B IT | 11.0 |
| Gemma 2 9B IT | 9.8 |
| Gemma 3 1B IT | 7.2 |
| Gemma 2 2B IT | 5.9 |