Atlas

Benchmarks

← All benchmarks

MultiPL-E (HumanEval, 8-language average)

Code · 2022-08-17

Average pass@1 over the MultiPL-E HumanEval translations in eight languages: Python, C++, Java, PHP, TypeScript, C#, Bash and JavaScript, the language set Qwen reports. MultiPL-E translates HumanEval problems and tests into other programming languages; averages over other language sets are not this benchmark.

Top models (higher is better)

ModelScore
o1 Preview85.3
o1-mini85.1
Claude 3.5 Sonnet (Oct 2024)83.8
Qwen2.5 Coder 14B Instruct79.6
Qwen2.5-VL-72B-Instruct79.5
Qwen2.5 Coder 32B Instruct79.4
GPT-4o (2024-08-06)79.1
Qwen2.5 14B Instruct 1M77.1
Qwen Plus (rolling alias)77.0
Mistral Large 2 (Instruct 2407)76.9
Qwen2.5-Coder-7B-Instruct76.5
Qwen2.5 Instruct 32B75.4
Qwen2.5 72B Instruct75.1
GPT-4o Mini75.0
Qwen2.5-Turbo73.7
Llama 3.1 405B Instruct73.5
DeepSeek-V2.573.5
DeepSeek-Coder-V2-Lite-Instruct73.2
Qwen-Turbo (rolling alias)73.0
Qwen2.5 14B Instruct72.8
Qwen2.5 7B Instruct 1M72.4
Qwen2.5 Coder 3B Instruct72.1
CodeQwen1.5 7B Chat70.8
Qwen2.5 7B Instruct70.4
Codestral 240570.2
Qwen2 72B Instruct69.2
Llama 3.1 70B Instruct68.2
Gemma 2 27B IT67.4
Qwen2-57B-A14B Instruct66.4
Qwen2.5-Coder-32B63.9
Llama 3 70B Instruct63.4
Mixtral 8x22B Instruct61.1
Qwen2.5 72B60.5
Qwen2.5 3B Instruct60.2
Qwen2.5-Coder-14B59.9
Qwen2-72B59.6
Qwen2.5 32B59.4
Qwen2 7B Instruct59.1
Qwen2.5 Coder 7B57.5
Qwen2.5 Coder 1.5B Instruct56.7
Loading Atlas data…