MBPP (pass@80)
Code · 2021-08-16
MBPP scored by pass@80, the metric PaLM and the LLaMA papers report: the estimated probability that at least one of 80 sampled programs passes a problem's assert-based tests, averaged over the 500-problem test split. It measures coverage under sampling rather than single-attempt accuracy.
Top models (higher is better)
| Model | Score |
|---|---|
| Llama 2 70B | 81.4 |
| LLaMA 65B | 76.8 |
| Llama 2 34B | 76.1 |
| PaLM 540B | 75.0 |
| LLaMA 33B | 73.4 |
| Llama 2 13B | 69.0 |
| LLaMA-13B | 64.0 |
| Llama 2 7B | 62.8 |
| LLaMA 7B | 56.2 |