MultiPL-E HumanEval (TypeScript)
Code · 2022-08-17
The TypeScript translation of HumanEval from MultiPL-E, which compiles the Python problems' signatures, docstrings and unit tests into other languages, dropping problems that cannot be translated. A problem is solved when the generated TypeScript code passes the translated tests; scores are pass@1.
Top models (higher is better)
| Model | Score |
|---|---|
| Llama 3.1 405B Instruct | 81.1 |
| Llama 3.1 70B Instruct | 73.0 |
| Llama 3.1 8B Instruct | 56.6 |
| Code Llama 70B | 49.1 |
| Code Llama Python 70B | 44.0 |
| Code Llama Instruct 34B | 40.3 |
| Code Llama Instruct 13B | 39.0 |
| Code Llama Instruct 70B | 39.0 |
| Code Llama Python 13B | 35.2 |
| Code Llama Python 34B | 34.3 |
| Code Llama 7B | 33.3 |
| Code Llama 34B | 33.3 |
| Code Llama Instruct 7B | 32.7 |
| Code Llama 13B | 29.6 |
| Code Llama Python 7B | 23.9 |
| Llama 2 34B | 21.4 |
| Llama 2 70B | 15.1 |
| Llama 2 13B | 13.2 |
| Llama 2 7B | 12.6 |