ZeroBench (Main questions, pass@1)

Name: ZeroBench (Main questions, pass@1)
Published: 2025-02-13

Multimodal · 2025-02-13

ZeroBench main-question single-sample pass@1 accuracy in percentage points. Kept distinct from the current five-sample mean pass@1 factor-group primary.

Top models (higher is better)

Model	Score
Doubao Seed 2.1 Pro	18.0
Qwen3.5 397B A17B	12.0
Seed2.1 Turbo	11.0
Qwen3.5-27B	10.0
Qwen3.5 122B A10B	9.0
Qwen3.5 35B A3B	8.0
Opus 4.1	5.0
GPT-5 Mini	4.0
Opus 4	4.0
Sonnet 4	3.0
o3	3.0
Gemini 2.5 Pro Experimental 03-25	3.0
Gemini 2.5 Flash Preview 04-17	3.0
Claude 3.7 Sonnet	3.0
Qwen3.5-9B	3.0
Qwen3.5-4B	3.0
GPT-5 Nano	2.0
Doubao 1.5 Thinking Vision Pro 250428	2.0
O4 Mini	2.0
GPT-5	1.0
Grok 4	1.0
GPT-4.5	1.0
Qwen3.5-2B	1.0
Sonnet 4.5	0.0
Llama 4 Maverick Instruct	0.0
Llama 4 Scout Instruct	0.0
GPT-4.1	0.0
O1 Pro	0.0
o1 (2024-12-17)	0.0
Gemini 2.0 Flash Thinking Experimental 01-21	0.0
QVQ 72B Preview	0.0
GPT-4o (2024-11-20)	0.0
GPT-4o Mini	0.0
Gemini 2.0 Flash	0.0
Gemini 1.5 Pro (source unspecified)	0.0
Gemini 1.5 Flash (source unspecified)	0.0
Gemini 1.0 Pro Vision	0.0
Claude 3.5 Sonnet (Oct 2024)	0.0
Claude 3.5 Sonnet (June 2024)	0.0
Opus 3	0.0