BBH (BIG-Bench Hard)
General QA · 2022-10-17
A curated subset of BIG-bench tasks that remain difficult for large language models, probing compositional, symbolic, and multi-step reasoning.
Top models (higher is better)
| Model | Score |
|---|---|
| Claude 3.5 Sonnet (Oct 2024) | 93.2 |
| Claude 3.5 Sonnet (June 2024) | 93.1 |
| Gemini 1.5 Pro (May 2024) | 89.2 |
| Opus 3 | 86.8 |
| Haiku 3.5 | 86.6 |
| Gemini 1.5 Flash (May 2024 report) | 85.5 |
| Gemini 1.0 Ultra | 83.6 |
| GPT-4 (source unspecified) | 83.1 |
| Claude 3 Sonnet | 82.9 |
| PaLM 2-L | 77.7 |
| Gemini 1.0 Pro | 75.0 |
| Claude 3 Haiku | 73.7 |
| Gemini Nano 2 | 42.4 |
| Gemini Nano 1 | 34.8 |