MBPP (pass@100)
Code · 2021-08-16
MBPP scored by pass@100: the estimated probability that at least one of 100 sampled programs passes a problem's assert-based tests, computed with the unbiased estimator and averaged over the 500-problem test split. It measures coverage under sampling rather than single-attempt accuracy.
Top models (higher is better)
| Model | Score |
|---|---|
| Code Llama 70B | 91.9 |
| Code Llama Python 70B | 91.9 |
| Code Llama Instruct 70B | 89.2 |
| Code Llama Python 34B | 88.2 |
| Code Llama Python 13B | 87.6 |
| Code Llama 13B | 87.1 |
| Code Llama 34B | 86.6 |
| Code Llama Instruct 34B | 85.4 |
| Code Llama Python 7B | 84.8 |
| Code Llama Instruct 13B | 84.1 |
| Code Llama 7B | 82.5 |
| Code Llama Instruct 7B | 76.8 |
| Llama 2 13B | 69.5 |
| Llama 2 7B | 65.5 |