ZeroEval CRUX

Name: ZeroEval CRUX
Published: 2024-08-01

Code · 2024-08-01

CRUX code reasoning evaluated with the ZeroEval zero-shot prompt format, greedy decoding, and answer parser.

Model	Score
o1 Preview	95.9
o1-mini	93.8
GPT-4o (2024-08-06)	87.0
ChatGPT-4o Latest (September 2024 benchmark entry)	86.5
GPT-4o (2024-05-13)	86.1
Claude 3.5 Sonnet (Oct 2024)	83.9
Claude 3.5 Sonnet (June 2024)	80.8
Gemini 1.5 Pro Experimental 0827	79.6
GPT-4 Turbo	78.9
GPT-4o Mini	75.9
Grok 2 1212	75.3
Mistral Large 2 (Instruct 2407)	75.1
Gemini 1.5 Pro Experimental 0801	74.9
GPT-4 (0314)	74.5
Gemini 1.5 Flash Experimental 0827	74.5
Llama 3.1 405B Instruct	74.1
Qwen2.5 72B Instruct	73.9
DeepSeek-V2-Chat-0628	70.5
Opus 3	70.4
DeepSeek-V2.5	70.0
Qwen2.5 Instruct 32B	69.9
DeepSeek-Coder-V2-Instruct-0724	69.5
Haiku 3.5	68.8
Gemini 1.5 Pro (source unspecified)	68.0
Claude 3 Sonnet	66.6
Llama 3.1 70B Instruct	64.3
Gemini 1.5 Flash (source unspecified)	63.8
Yi-Large	60.4
Qwen2 72B Instruct	59.1
Llama 3 70B Instruct	58.9
Gemma 2 27B IT	57.3
Claude 3 Haiku	54.8
GPT-3.5 Turbo 0125	54.8
Qwen2.5 7B Instruct	52.8
Athene-70B	50.6
Reka Core 20240501 (ZeroEval label)	46.3
Gemma 2 9B IT	46.0
Mixtral 8x7B Instruct	44.9
Phi-3-mini-4k-instruct	44.8
Yi 1.5 9B Chat	44.8