Arena-Hard-Auto v2.0

Name: Arena-Hard-Auto v2.0
Published: 2025-04-23

Chat & Writing · 2025-04-23

Arena-Hard-Auto v2.0 is LMArena's second automatic chat benchmark, 500 hard prompts and 250 creative-writing prompts mined from Chatbot Arena, each answer compared with a fixed per-category baseline by an LLM judge and scored as a win rate. This row is the single headline win rate that Qwen's releases report with a GPT-4.1 judge, which does not say how the two categories are combined; it is the family's primary, and the hard-prompts, creative-writing and averaged figures that NVIDIA and Moonshot report separately are its components. Higher is better.

Top models (higher is better)

Model	Score
Qwen3 Max Thinking (2026-01-23)	90.2
Qwen3 Next 80B A3B Instruct	82.7
Gemini 3 Pro Preview	81.7
o3	80.8
GPT-5.2	80.6
Qwen3 235B A22B Thinking 2507	79.7
Qwen3 235B A22B Instruct 2507	79.2
Qwen3-VL-235B-A22B-Instruct	77.4
Opus 4.5	76.7
Qwen3 VL 235B A22B Thinking	74.8
Gemini 2.5 Pro	72.5
DeepSeek-R1-0528	72.2
Qwen3 30B A3B Instruct 2507	69.0
DeepSeek-V3.2	66.5
Kimi K2 Instruct	66.1
Qwen3 VL 32B Instruct	64.7
Qwen3-Next-80B-A3B-Thinking	62.3
ChatGPT-4o Latest observed 2025-03-27	61.9
Qwen3-235B-A22B	61.5
Qwen3 VL 32B Thinking	60.5
O4 Mini	59.3
Opus 4	59.1
Qwen3 VL 30B A3B Instruct	58.5
Gemini 2.5 Flash	58.3
Qwen3 VL 30B A3B Thinking	56.7
Qwen3 30B A3B Thinking 2507	56.0
Qwen3 Omni 30B A3B Thinking	55.1
Qwen3 VL 8B Thinking	51.1
Qwen3-VL-8B-Instruct	46.3
DeepSeek-V3-0324	45.6
Qwen3 4B Instruct 2507	43.4
Qwen3 VL 4B Thinking	36.8
Qwen3-30B-A3B	36.3
Qwen3 4B Thinking 2507	34.9
Qwen3 VL 4B Instruct	30.4
Qwen3 8B	29.1
GPT-4.1 Nano	15.9
Qwen3 4B	13.7
Qwen3 VL 2B Thinking	12.0
Qwen3 VL 2B Instruct	6.4