SimpleVQA - release-reported, metric unspecified
Multimodal · 2025-02-18
SimpleVQA is a 2,025-question English and Chinese factuality VQA benchmark whose short answers an LLM judge (GPT-4o canonically) grades as correct, incorrect or not attempted, reported as a correct rate and an F-score. Release tables that print one SimpleVQA number without saying which metric it is, often with a different judge, are kept as a diagnostic; the judge is recorded on each score.
Top models (higher is better)
| Model | Score |
|---|---|
| Gemini 3.5 Flash-Lite | 56.9 |
| GPT-5.6 Luna | 53.2 |
| Qwen3.5-4B | 45.8 |
| Gemma 4 12B IT | 42.5 |
| Qwen3.5-2B | 38.0 |
| LFM3-VL-2B | 36.3 |
| LFM2.5-VL-3B | 35.4 |
| Gemma 4 E2B IT | 28.4 |