SimpleVQA
Multimodal · 2025-02-18
SimpleVQA measures multimodal factuality with 2,025 short, verifiable image-grounded questions in English and Chinese whose answers do not change over time. An LLM judge grades each response correct, incorrect or not attempted, and the score is the correct rate over all questions without search tools; search-augmented runs are the separate simplevqa-with-tools benchmark. Higher is better.
Top models (higher is better)
| Model | Score |
|---|---|
| Qwen3.8-Max | 75.0 |
| Gemini 3 Pro Preview | 73.2 |
| Gemini 3.1 Pro Preview | 73.1 |
| Muse Spark | 71.3 |
| Kimi K2.5 | 71.2 |
| Qwen3.7-Plus (source unspecified) | 70.3 |
| Qwen3.6 Plus (2026-04-02) | 67.3 |
| Opus 4.8 | 67.3 |
| Qwen3.5 397B A17B | 67.1 |
| GPT-5.6 Sol | 66.6 |
| Qwen3.5 Plus (2026-02-15) | 66.1 |
| Opus 4.5 | 65.7 |
| Qwen3.5-Omni-Plus | 65.3 |
| Opus 4.6 | 62.2 |
| Qwen3.5 122B A10B | 61.7 |
| GPT-5.4 | 61.1 |
| Qwen3.6 35B A3B | 58.9 |
| Qwen3.5 35B A3B | 58.3 |
| Sonnet 4.5 | 57.6 |
| Grok 4.20 | 57.4 |
| GPT-5 Mini | 56.8 |
| Qwen3.6 27B | 56.1 |
| Qwen3.5-27B | 56.0 |
| GPT-5.2 | 55.8 |
| Qwen3.5-Omni-Flash | 54.4 |
| Gemini 2.5 Flash-Lite | 54.1 |
| Gemma 4 31B IT | 52.9 |
| Gemma 4 26B A4B IT | 52.2 |
| Qwen3.5-9B | 51.2 |
| GPT-5 Nano | 46.0 |
| Qwen3.5-4B | 43.4 |
| Qwen3.5-2B | 39.5 |
| Qwen3.5-0.8B | 31.3 |