MBPP (pass@10)
Code · 2021-08-16
MBPP scored by pass@10: the estimated probability that at least one of ten sampled programs passes a problem's three assert-based tests, computed with the unbiased estimator and averaged over the 500-problem test split. It uses temperature sampling, so it is kept apart from pass@1.
Top models (higher is better)
| Model | Score |
|---|---|
| Code Llama Python 70B | 81.5 |
| Code Llama 70B | 81.1 |
| Code Llama Instruct 70B | 79.6 |
| Code Llama Python 34B | 76.4 |
| Code Llama 34B | 76.2 |
| Code Llama Instruct 34B | 74.6 |
| Code Llama Python 13B | 74.0 |
| Code Llama 13B | 71.7 |
| Code Llama Instruct 13B | 71.2 |
| Code Llama Python 7B | 70.3 |
| Code Llama 7B | 66.7 |
| Llama 2 70B | 66.2 |
| Code Llama Instruct 7B | 65.4 |
| Llama 2 34B | 56.9 |
| Llama 2 13B | 48.1 |
| Llama 2 7B | 41.8 |