CountBench
Multimodal · 2023-02-23
CountBench is a set of web image-caption pairs whose captions state an object count from two to ten, used as a counting question-answering test (CountBenchQA) for multimodal models. Scores report the share of images where the model gives the correct count.
Top models (higher is better)
| Model | Score |
|---|---|
| Qwen3.5-27B | 97.8 |
| Qwen3.5 35B A3B | 97.8 |
| Qwen3.6 27B | 97.8 |
| Qwen3.6 Plus (2026-04-02) | 97.6 |
| Gemini 3 Pro Preview | 97.3 |
| Qwen3.5 397B A17B | 97.2 |
| Qwen3.5-9B | 97.2 |
| Qwen3.5 122B A10B | 97.0 |
| Qwen3.5-4B | 96.3 |
| Gemma 4 31B IT | 96.1 |
| Qwen3.6 35B A3B | 96.1 |
| Qwen3.5 Plus (2026-02-15) | 95.1 |
| Qwen3.5-Omni-Plus | 95.1 |
| Qwen3 VL 32B Instruct | 94.9 |
| Kimi K2.5 | 94.1 |
| Qwen3 VL 32B Thinking | 94.1 |
| Qwen3 VL 235B A22B Thinking | 93.7 |
| Qwen2.5-VL-72B-Instruct | 93.6 |
| Opus 4.1 | 93.1 |
| Qwen3-VL-235B-A22B-Instruct | 93.0 |
| GPT-5.2 | 91.9 |
| GPT-5 | 91.7 |
| Qwen3 VL 8B Thinking | 91.5 |
| Sonnet 4 | 91.5 |
| Qwen3.5-2B | 91.4 |
| Gemini 2.0 Flash | 91.2 |
| GPT-5 Mini | 91.0 |
| Gemini 2.5 Pro | 91.0 |
| Opus 4.5 | 90.6 |
| Sonnet 4.5 | 90.0 |
| Qwen3 Omni 30B A3B Instruct | 90.0 |
| Qwen3 VL 30B A3B Thinking | 90.0 |
| Qwen3 VL 30B A3B Instruct | 89.8 |
| Qwen3 VL 4B Thinking | 89.4 |
| Qwen3 Omni 30B A3B Thinking | 88.6 |
| Qwen3 VL 2B Instruct | 88.4 |
| Qwen3.5-Omni-Flash | 88.2 |
| Qwen3 VL 4B Instruct | 84.9 |
| Qwen3 VL 2B Thinking | 84.1 |
| Gemini 2.5 Flash | 83.7 |