Arena-Hard-Auto v0.1

Name: Arena-Hard-Auto v0.1
Published: 2024-04-19

Chat & Writing · 2024-04-19

Arena-Hard-Auto v0.1 is LMSYS's automatic chat benchmark of 500 challenging user prompts selected from Chatbot Arena. Each answer is compared pairwise with GPT-4-0314's by a GPT-4-Turbo (gpt-4-1106-preview) judge, and the score is the model's estimated win rate against that baseline in percent, without style control; rows whose source does not restate the judge are noted. Arena-Hard v2.0 and runs with a non-official judge (arena-hard-reported) are separate. Higher is better.

Top models (higher is better)

Model	Score
Qwen3-235B-A22B	96.1
Qwen3 32B	93.8
o1 (2024-12-17)	92.1
o1-mini	92.0
Qwen3 14B	91.7
Qwen3-30B-A3B	91.0
Qwen2.5-Max	89.4
o3-mini	89.0
Llama 3.3 Nemotron Super 49B V1	88.3
Llama 3.1 Nemotron Ultra 253B V1	87.0
Gemma 3 27B IT	86.8
Qwen3 8B	85.8
DeepSeek-V3	85.5
GPT-4o (2024-11-20)	85.3
Claude 3.5 Sonnet (Oct 2024)	85.2
Llama 3.1 Nemotron 70B Instruct HF	85.0
Llama 4 Maverick Instruct	82.7
Gemma 3 12B IT	82.6
Qwen Plus (rolling alias)	81.4
Qwen2.5 72B Instruct	81.2
GPT-4o (2024-08-06)	77.9
Qwen3 4B	76.6
DeepSeek-V2.5	76.2
Phi-4	75.4
GPT-4o Mini	74.9
Qwen2.5 Instruct 32B	74.5
Mistral Large 2 (Instruct 2407)	73.1
Llama 3.3 70B Instruct	72.9
Llama 4 Scout Instruct	70.5
Qwen2.5 14B Instruct 1M	70.2
Qwen-Turbo (rolling alias)	68.4
DeepSeek-V2-Chat-0628	68.3
Qwen2.5 14B Instruct	68.3
Qwen2.5-Turbo	67.1
DeepSeek-Coder-V2-Instruct-0724	66.3
DeepSeek-Coder-V2-Instruct	65.0
DeepSeek-R1-Distill-Qwen-32B	60.8
Gemma 2 27B IT	57.5
Llama 3.1 70B Instruct	55.7
Nemotron-4 340B Instruct	54.2