CRUXEval-I
Code · 2024-01-05
CRUXEval-I is the input-prediction task of CRUXEval: given one of 800 short Python functions and an output, the model must produce an input that makes the function return that output, checked by execution. Scores are the share of functions solved (pass@1, which DeepSeek tables label EM); chain-of-thought prompting is a protocol detail recorded in score notes. Higher is better.
Top models (higher is better)
| Model | Score |
|---|---|
| GPT-4o (2024-05-13) | 77.4 |
| GPT-4 Turbo | 75.7 |
| GPT-4 Turbo (1106 Preview) | 75.5 |
| Opus 3 | 73.4 |
| DeepSeek-Coder-V2-Instruct | 70.0 |
| DeepSeek-V3-Base | 67.3 |
| DeepSeek-V2-Chat | 61.5 |
| Llama 3 70B Instruct | 61.1 |
| Mixtral 8x22B Instruct | 59.4 |
| Qwen2.5 72B | 59.1 |
| Llama 3.1 405B | 58.5 |
| DeepSeek-Coder-V2-Lite-Instruct | 53.0 |
| Mixtral 8x22B | 52.4 |
| Qwen1.5 72B Chat | 51.4 |
| Llama 3 70B | 49.4 |
| DeepSeek LLM 67B Chat | 49.1 |
| Codestral 2405 | 48.0 |
| DeepSeek-Coder-Instruct 33B | 47.3 |
| Qwen1.5 72B | 44.3 |
| DeepSeek LLM 67B Base | 42.5 |