Atlas

Benchmarks

← All benchmarks

HumanEval-Mul

Code · 2024-01-25

HumanEval-Mul is DeepSeek's multilingual HumanEval: the 164 HumanEval problems translated into eight languages (Python, Java, C++, C#, JavaScript, TypeScript, PHP and Bash), first reported as multilingual HumanEval in the DeepSeek-Coder paper. The score is pass@1 averaged over the languages, so it is a mean of per-language rates. Higher is better.

Top models (higher is better)

ModelScore
DeepSeek-V382.6
Claude 3.5 Sonnet (Oct 2024)81.7
GPT-4o (2024-05-13)80.5
Qwen2.5 72B Instruct77.3
Llama 3.1 405B Instruct77.2
DeepSeek-Coder-V2-Instruct-072474.8
DeepSeek-V2-Chat-062873.8
DeepSeek-V2-Chat69.3
DeepSeek-Coder-Instruct 33B69.2
DeepSeek-Coder-Instruct 6.7B66.1
DeepSeek-Coder-Instruct 7B v1.564.1
DeepSeek LLM 67B Chat53.3
DeepSeek-Coder-Base 33B50.3
DeepSeek-Coder-Instruct 1.3B48.4
DeepSeek-Coder-Base 6.7B44.7
DeepSeek-Coder-Base 7B v1.543.2
DeepSeek LLM 67B Base37.2
DeepSeek-Coder-Base 1.3B28.3
Loading Atlas data…