Atlas

Benchmarks

← All benchmarks

HumanEval+

Code · 2023-05-02

HumanEval+ is EvalPlus's extension of OpenAI's 164-problem HumanEval Python benchmark, re-scoring each generated solution against roughly 80 times more test inputs than the original suite to catch solutions that only pass the sparse original tests. Scores report pass@1, the share of problems whose solution passes every extended test. Higher is better.

Top models (higher is better)

ModelScore
o1-mini90.2
o1 Preview88.4
Qwen2.5 Coder 14B Instruct87.2
Qwen2.5 Coder 32B Instruct87.2
GPT-4o (source unspecified)86.0
Claude 3.5 Sonnet (Oct 2024)86.0
GPT-4o (2024-08-06)86.0
GPT-4o Mini84.8
Qwen2.5-Coder-7B-Instruct84.1
Llama 3.1 405B Instruct82.3
GPT-4 Turbo (1106 Preview)81.7
Qwen2.5 Coder 3B Instruct80.5
CodeQwen1.5 7B Chat78.7
GPT-4 0125 Preview77.4
DeepSeek-Coder-V2-Lite-Instruct75.6
Llama 3.1 70B Instruct74.4
Codestral 240573.2
Kimi Linear 48B A3B Instruct70.9
Nemotron-H 4B Instruct 128K70.8
Qwen2.5 3B Instruct70.1
Mixtral 8x22B Instruct68.3
Llama 3.1 8B Instruct67.1
Qwen2.5 Coder 1.5B Instruct66.5
Code Llama Instruct 70B65.9
Phi-4-mini-instruct64.6
Nemotron Nano 12B V2 Base61.0
Qwen2.5-Coder-32B60.4
Nemotron Nano 9B V2 Base58.5
Qwen2.5-Coder-14B57.9
Qwen2.5 Coder 0.5B Instruct57.3
Nemotron-H 47B Base 8K56.1
Qwen2-72B56.1
Nemotron-H 4B Base 8K55.5
Qwen2.5 Coder 7B53.0
Llama 3.1 70B52.4
Qwen2.5 32B52.4
Llama 3.2 3B Instruct51.8
Qwen2.5 14B51.2
Qwen2.5 7B50.6
Gemma 2 9B IT48.8
Loading Atlas data…