HumanEval (pass@10)
Code · 2021-07-07
HumanEval scored by pass@10: the estimated probability that at least one of ten sampled programs passes a problem's hidden unit tests, computed with the unbiased estimator from n samples per problem and averaged over the 164 problems. It uses temperature sampling rather than greedy decoding, so it is kept apart from pass@1.
Top models (higher is better)
| Model | Score |
|---|---|
| Code Llama Instruct 70B | 90.3 |
| Code Llama Python 70B | 89.3 |
| Code Llama 70B | 84.6 |
| Code Llama Python 34B | 82.8 |
| Code Llama Python 13B | 77.4 |
| Code Llama Instruct 34B | 77.2 |
| Code Llama 34B | 76.8 |
| Code Llama Instruct 13B | 71.6 |
| Code Llama Python 7B | 70.3 |
| Code Llama 13B | 69.4 |
| Code Llama Instruct 7B | 64.3 |
| Code Llama 7B | 59.6 |
| Llama 2 70B | 59.4 |
| Llama 2 34B | 47.0 |
| Llama 2 13B | 34.8 |
| Llama 2 7B | 25.2 |