VLMs Are Blind
Multimodal · 2024-07-09
VLMs Are Blind tests low-level visual perception with simple synthetic tasks that humans solve almost perfectly, such as counting line intersections, judging whether circles overlap, identifying circled letters and counting nested shapes or grid rows. Scores report mean accuracy across the tasks.
Top models (higher is better)
| Model | Score |
|---|---|
| Qwen3.5 35B A3B | 97.0 |
| Qwen3.6 27B | 97.0 |
| Qwen3.5-27B | 96.9 |
| Qwen3.5 122B A10B | 96.7 |
| Qwen3.6 35B A3B | 96.6 |
| Qwen3.5-9B | 93.7 |
| Qwen3.5-4B | 92.6 |
| o3 | 90.1 |
| O4 Mini | 87.3 |
| Gemma 4 31B IT | 87.2 |
| Sonnet 4.5 | 85.5 |
| GPT-5 Mini | 75.8 |
| Qwen3.5-2B | 75.8 |
| Gemini 2.5 Flash-Lite | 68.4 |
| GPT-5 Nano | 66.7 |
| Qwen3.5-0.8B | 59.4 |
| o1 (2024-12-17) | 57.0 |
| GPT-4o (source unspecified) | 50.4 |