ZebraLogic

Name: ZebraLogic
Published: 2025-02-03

Math · 2025-02-03

ZebraLogic is a set of 1,000 logic-grid (Einstein) puzzles of controlled size and difficulty; a puzzle counts as solved only when every cell of the solution grid is correct. This row is the benchmark as run by a model developer with its own prompting and sampling, distinct from the greedy ZeroEval leaderboard protocol, with which it shares a factor group.

Top models (higher is better)

Model	Score
Qwen3 235B A22B Thinking 2507	98.1
Qwen3 VL 235B A22B Thinking	97.3
o3	95.8
Opus 4	95.1
Qwen3 235B A22B Instruct 2507	95.0
Qwen3 30B A3B Instruct 2507	90.0
Qwen3-30B-A3B	89.5
Kimi K2 Instruct	89.0
o3-mini	88.9
Qwen3 32B	88.8
Qwen3 14B	88.5
Qwen3 8B	84.8
DeepSeek-V3-0324	83.4
Qwen3 4B	81.0
o1 (2024-12-17)	81.0
Qwen3-235B-A22B	80.3
Qwen3 4B Instruct 2507	80.2
DeepSeek-R1	78.7
QwQ-32B	76.8
Qwen3 Omni 30B A3B Instruct	76.0
DeepSeek-R1-Distill-Llama-70B	71.3
DeepSeek-R1-Distill-Qwen-32B	69.6
Qwen3 1.7B	63.2
DeepSeek R1 Distill Qwen 14B	59.1
Gemini 2.5 Flash	57.9
ChatGPT-4o Latest observed 2025-03-27	52.6
DeepSeek-V3	42.1
Llama 4 Maverick Instruct	40.0
DeepSeek R1 Distill Llama 8B	37.1
Phi-4	32.3
Qwen3 0.6B	30.3
GPT-4o (2024-11-20)	27.4
Qwen2.5 72B Instruct	26.6
Qwen2.5 Instruct 32B	26.1
Gemma 3 27B IT	24.6
Llama 4 Scout Instruct	24.2
GPT-4o Mini	20.1
Qwen2.5 14B Instruct	19.7
Gemma 3 12B IT	17.8
GPT-4.1 Nano	14.8