MultiPL-E HumanEval (Bash)

Name: MultiPL-E HumanEval (Bash)
Published: 2022-08-17

Code · 2022-08-17

The Bash translation of HumanEval from MultiPL-E, which compiles the Python problems' signatures, docstrings and unit tests into other languages, dropping problems that cannot be translated. A problem is solved when the generated Bash code passes the translated tests; scores are pass@1.

Top models (higher is better)

Model	Score
Llama 3.1 405B Instruct	57.6
Llama 3.1 70B Instruct	51.9
Llama 3.1 8B Instruct	39.2
Code Llama Instruct 70B	29.7
Code Llama 70B	29.1
Code Llama Python 70B	25.3
Code Llama Instruct 34B	19.6
Code Llama 34B	17.1
Code Llama Python 7B	16.5
Code Llama 13B	15.2
Code Llama Python 34B	14.6
Code Llama Instruct 13B	13.9
Code Llama Python 13B	13.9
Code Llama 7B	12.0
Code Llama Instruct 7B	10.1
Llama 2 70B	8.9
Llama 2 34B	3.8
Llama 2 7B	3.2
Llama 2 13B	3.2