Curator Evals: Code Correctness
Code · 2025-09-02
Collinear AI's benchmark of models acting as post-training data curators. Given a coding problem with a candidate solution, or a HumanEvalPack correctness preference pair, the model must answer only with a 1-or-0 correctness verdict. The 1,302 problems come from HumanEvalPack's correctness pairs and a subset of MBPP whose solutions are checked against its tests. The score is the leaderboard's accuracy in percent; the launch post gives no formula, label balance, split sizes or sampling settings, and the dataset is gated. Higher is better.
Top models (higher is better)
| Model | Score |
|---|---|
| Qwen2.5-Coder-7B-Instruct | 76.9 |
| GPT-4o (source unspecified) | 63.7 |
| DeepSeek R1 0528 Qwen3 8B | 63.7 |
| Qwen3 8B | 60.6 |
| Qwen2.5 Coder 3B Instruct | 46.8 |