MBPP EvalPlus (base tests)

Name: MBPP EvalPlus (base tests)
Published: 2023-05-02

Code · 2023-05-02

The EvalPlus sanitized subset of MBPP problems, scored against the original MBPP base tests rather than EvalPlus's extended MBPP+ test suites. Meta reports it as "MBPP EvalPlus (base)" with 0-shot pass@1. It is kept separate from MBPP+ (extended tests) and from the 500-problem MBPP test split.

Top models (higher is better)

Model	Score
Claude 3.5 Sonnet (June 2024)	90.5
Llama 3.1 405B Instruct	88.6
GPT-4o (source unspecified)	87.8
Llama 3.3 70B Instruct	87.6
Llama 3.1 70B Instruct	86.0
GPT-4 0125 Preview	83.6
Llama 3 70B Instruct	82.5
Mixtral 8x22B Instruct	78.6
Llama 3.1 8B Instruct	72.8
Gemma 2 9B IT	71.7
Llama 3 8B Instruct	70.6