Atlas

Benchmarks

← All benchmarks

MBPP

Code · 2021-08-16

MBPP (Mostly Basic Python Problems) is Google Research's set of 974 crowd-sourced entry-level Python programming tasks, each given as a short natural-language description with three assert-based unit tests. Its canonical evaluation is the 500-problem test split (task IDs 11-510), usually prompted with three solved examples and scored as pass@1; some release reports use the 427-problem sanitized subset or EvalPlus's base-test set instead, and each row's notes record the split and shots. MBPP+ with EvalPlus's extended tests is the separate mbpp-plus benchmark. Higher is better.

Top models (higher is better)

ModelScore
o1-mini93.9
o1 Preview93.4
Nemotron-H 47B Reasoning 128K91.8
Claude 3.5 Sonnet (Oct 2024)91.0
Qwen2.5 Coder 32B Instruct90.2
Qwen2.5 72B Instruct88.2
GPT-4o (2024-08-06)86.8
Qwen2.5 Coder 14B Instruct86.2
Nemotron-H 8B Reasoning 128K86.0
Nemotron 3 Ultra 550B A55B Base86.0
Llama 3.1 Nemotron Nano 4B V1.185.8
Haiku 3.585.6
Qwen Plus (rolling alias)85.5
DeepSeek-V2.585.2
Llama 3.3 Nemotron Super 49B V184.9
Qwen2.5 32B84.5
Qwen2.5 VL 32B Instruct84.0
Qwen2.5 Instruct 32B84.0
Qwen2.5-Coder-7B-Instruct83.5
Qwen2.5-Turbo82.8
DeepSeek-Coder-V2-Lite-Instruct82.8
Qwen2.5 14B Instruct82.0
GPT-4o (source unspecified)81.4
Codestral 25.0180.2
GPT-4 0125 Preview80.2
Qwen2 72B Instruct80.2
Qwen2.5 14B Instruct 1M80.2
Qwen-Turbo (rolling alias)80.2
Mistral Large 2 (Instruct 2407)80.0
Qwen2.5 7B Instruct79.2
Nemotron-H 4B Instruct 128K78.6
Llama 4 Maverick77.6
Qwen2-72B76.9
Qwen2.5 14B76.7
Qwen1.5 110B Chat76.4
Nemotron-H 47B Base 8K75.9
Qwen2.5 7B Instruct 1M75.9
DeepSeek-V3-Base75.4
Nemotron-4 340B Instruct75.4
Qwen2.5 7B74.9
Loading Atlas data…