Turing VLMBench - Hard Subset

Multimodal

Turing VLMBench's hard subset: the 189 questions Turing identifies as the most challenging of its 705, scored from the same three judged answers per question. Turing does not describe how the subset was selected.

Top models (higher is better)

Model	Score
O4 Mini	10.1
Sonnet 4	7.4
Opus 4	6.7
Gemini 2.0 Flash	6.1
GPT-4.1 Mini	5.6
Claude 3.7 Sonnet	5.5
Llama 4 Maverick Instruct	4.8
o3	4.8
o1 (source unspecified)	3.4
GPT-4.1	3.3
Llama 4 Scout Instruct	3.3
GPT-4.1 Nano	3.0
GPT-4o (2024-11-20)	3.0
GPT-4o (2024-08-06)	2.9
Llama 3.2 11B Vision Instruct	2.9
Llama 3.2 90B Vision Instruct	2.7
GPT-4o Mini	2.2