CountBenchQA
Multimodal · 2024-07-10
CountBenchQA turns CountBench's counting image set into visual question answering: each image shows 2 to 10 instances of an object and the model must say how many. Google introduced it with PaliGemma and scores exact-match accuracy on the full split; released copies hold about 490 usable images, so the item count is left unset.
Top models (higher is better)
| Model | Score |
|---|---|
| Gemma 4 12B IT | 95.9 |
| LFM3-VL-2B | 93.0 |
| Qwen3.5-4B | 92.6 |
| GPT-5.6 Luna | 91.0 |
| Gemini 3.5 Flash-Lite | 90.1 |
| LFM2.5-VL-3B | 87.3 |
| Qwen3.5-2B | 87.1 |
| Gemma 4 E2B IT | 78.2 |
| LFM2.5-VL-450M | 73.3 |
| LFM2-VL-450M | 47.6 |