CRUXEval-O
Code · 2024-01-05
CRUXEval-O is the output-prediction task of CRUXEval: given one of 800 short Python functions and an input, the model must predict the function's output, checked against execution. Scores are the share of functions solved (pass@1, which DeepSeek tables label EM); chain-of-thought prompting is a protocol detail recorded in score notes. Higher is better.
Top models (higher is better)
| Model | Score |
|---|---|
| GPT-4o (2024-05-13) | 88.7 |
| Opus 3 | 82.0 |
| GPT-4 Turbo | 82.0 |
| GPT-4 Turbo (1106 Preview) | 77.1 |
| DeepSeek-Coder-V2-Instruct | 75.1 |
| DeepSeek-V3-Base | 69.8 |
| Mixtral 8x22B Instruct | 63.6 |
| DeepSeek-V2-Chat | 63.0 |
| Codestral 2405 | 60.6 |
| Qwen2.5 72B | 59.9 |
| Llama 3.1 405B | 59.9 |
| Qwen1.5 72B Chat | 56.5 |
| Llama 3 70B | 54.3 |
| DeepSeek-Coder-V2-Lite-Instruct | 52.9 |
| Mixtral 8x22B | 52.8 |
| DeepSeek LLM 67B Chat | 50.9 |
| DeepSeek-Coder-Instruct 33B | 50.6 |
| DeepSeek-V2 | 49.8 |
| Qwen1.5 72B | 42.3 |
| DeepSeek LLM 67B Base | 41.0 |