CRUXEval-O (CoT)
Code · 2024-01-05
CRUXEval-O is the output-prediction half of CRUXEval: for each of 800 short Python functions and a given input, the model must predict the function's output, here with chain-of-thought reasoning allowed before the answer. Scores report pass@1 accuracy. Higher is better.
Top models (higher is better)
| Model | Score |
|---|---|
| Kimi K2 Base | 83.5 |
| Llama 4 Maverick | 75.9 |
| DeepSeek-V3-Base | 75.3 |
| Kimi Linear 48B A3B Base | 67.0 |
| Qwen2.5 72B | 66.1 |
| Moonlight 16B A3B | 46.6 |