HumanEval (pass@100)
Code · 2021-07-07
HumanEval scored by pass@100: the estimated probability that at least one of 100 sampled programs passes a problem's hidden unit tests, computed with the unbiased estimator from n samples per problem and averaged over the 164 problems. It measures coverage under sampling rather than single-attempt accuracy.
Top models (higher is better)
| Model | Score |
|---|---|
| Code Llama Python 70B | 98.4 |
| Code Llama Instruct 70B | 97.3 |
| Code Llama 70B | 96.2 |
| Code Llama Python 34B | 94.7 |
| Code Llama Python 13B | 94.1 |
| Code Llama Instruct 34B | 93.5 |
| Code Llama 34B | 93.0 |
| Code Llama Instruct 13B | 91.6 |
| Code Llama Python 7B | 90.6 |
| Code Llama 13B | 89.8 |
| Code Llama Instruct 7B | 88.1 |
| Code Llama 7B | 85.9 |
| LLaMA 65B | 79.3 |
| PaLM 540B | 76.2 |
| LLaMA 33B | 70.7 |
| LLaMA-13B | 52.5 |
| LLaMA 7B | 36.5 |