CRUXEval-I (CoT)
Code · 2024-01-05
CRUXEval-I is the input-prediction half of CRUXEval: for each of 800 short Python functions and a given output, the model must produce an input that yields that output, here with chain-of-thought reasoning allowed before the answer. Scores report pass@1 accuracy. Higher is better.
Top models (higher is better)
| Model | Score |
|---|---|
| Kimi K2 Base | 74.0 |
| Llama 4 Maverick | 67.1 |
| DeepSeek-V3-Base | 62.8 |
| Qwen2.5 72B | 61.1 |
| Kimi Linear 48B A3B Base | 61.0 |
| Moonlight 16B A3B | 45.9 |