MultiPL-E HumanEval (C#)

Name: MultiPL-E HumanEval (C#)
Published: 2022-08-17

Code · 2022-08-17

The C# translation of HumanEval from MultiPL-E, which compiles the Python problems' signatures, docstrings and unit tests into other languages, dropping problems that cannot be translated. A problem is solved when the generated C# code passes the translated tests; scores are pass@1.

Top models (higher is better)

Model	Score
Llama 3.1 405B Instruct	54.4
Llama 3.1 70B Instruct	50.0
Code Llama 70B	38.0
Llama 3.1 8B Instruct	38.0
Code Llama Instruct 70B	36.7
Code Llama Python 70B	34.8
Code Llama Python 34B	31.7
Code Llama Instruct 34B	31.0
Code Llama 34B	30.4
Code Llama Python 13B	29.8
Code Llama 13B	27.3
Llama 2 70B	25.9
Code Llama 7B	25.3
Code Llama Python 7B	24.7
Code Llama Instruct 13B	24.0
Code Llama Instruct 7B	21.6
Llama 2 34B	17.1
Llama 2 13B	9.5
Llama 2 7B	6.3