ZeroBench (Subquestions, pass@1)
Multimodal · 2025-02-13
This ZeroBench split evaluates 334 component subquestions grouped under 100 main visual-reasoning questions. The official score first computes accuracy within each main-question group and then takes the unweighted mean across the 100 groups, so the published n_correct micro-count need not equal score × 334.
Top models (higher is better)
| Model | Score |
|---|---|
| Doubao Seed 2.1 Pro | 49.4 |
| Seed2.1 Turbo | 49.1 |
| Doubao Seed 2.0 Pro | 47.6 |
| Doubao Seed 2.0 Lite | 42.2 |
| Gemini 3 Pro Preview | 42.2 |
| Gemini 3.1 Pro Preview | 41.9 |
| Qwen3.5 397B A17B | 41.0 |
| Qwen3.5-27B | 36.2 |
| Qwen3.5 122B A10B | 36.2 |
| Qwen3.5 35B A3B | 34.1 |
| Qwen3.5-9B | 31.1 |
| Doubao 1.5 Thinking Vision Pro 250428 | 30.8 |
| O4 Mini | 29.1 |
| GPT-5 Mini | 27.8 |
| GPT-4.5 | 27.0 |
| Qwen3.5-4B | 26.3 |
| GPT-5 | 26.2 |
| Gemini 2.5 Pro Experimental 03-25 | 26.0 |
| o3 | 25.5 |
| Claude 3.5 Sonnet (Oct 2024) | 25.5 |
| Opus 4.1 | 25.3 |
| Opus 4 | 25.1 |
| Sonnet 4 | 24.6 |
| Gemini 2.5 Flash Preview 04-17 | 24.5 |
| Llama 4 Maverick Instruct | 23.7 |
| Gemini 2.0 Flash | 23.2 |
| O1 Pro | 22.4 |
| GPT-5 Nano | 21.6 |
| Grok 4 | 21.6 |
| Gemini 1.5 Pro (source unspecified) | 20.9 |
| Claude 3.5 Sonnet (June 2024) | 20.7 |
| GPT-4.1 | 20.7 |
| Gemini 2.0 Flash Thinking Experimental 01-21 | 20.5 |
| QVQ 72B Preview | 20.5 |
| o1 (2024-12-17) | 20.2 |
| GPT-4o (2024-11-20) | 19.6 |
| Claude 3.7 Sonnet | 19.0 |
| Pixtral Large | 18.7 |
| Qwen3.5-2B | 18.6 |
| Gemini 1.5 Flash (source unspecified) | 17.9 |