Turing VLMBench
Multimodal
Turing's VLMBench: 705 private, open-ended visual questions written by business and STEM professionals across 23 subject areas (finance 129, math 91, marketing and sales 47, and engineering, business and science fields), each pairing an image such as a chart or technical diagram with a practical task for a professional persona. Each model answers every question three times, an unnamed LLM judge with rubrics scores each answer from 0 to 1, a question scores the mean of its three answers, and the score is the mean over questions. Turing reports that human judges disagreed with the judge on 1% of 300 checked examples.
Top models (higher is better)
| Model | Score |
|---|---|
| O4 Mini | 40.6 |
| Claude 3.7 Sonnet | 35.3 |
| Sonnet 4 | 32.6 |
| GPT-4.1 Mini | 32.6 |
| Opus 4 | 32.3 |
| Gemini 2.0 Flash | 29.0 |
| o3 | 26.2 |
| Llama 4 Maverick Instruct | 22.0 |
| o1 (source unspecified) | 19.6 |
| GPT-4.1 | 17.5 |
| Llama 4 Scout Instruct | 17.0 |
| GPT-4.1 Nano | 16.2 |
| GPT-4o (2024-11-20) | 14.1 |
| GPT-4o (2024-08-06) | 13.1 |
| Llama 3.2 90B Vision Instruct | 9.7 |
| GPT-4o Mini | 9.2 |
| Llama 3.2 11B Vision Instruct | 6.4 |