ZeroEval GSM8K

Name: ZeroEval GSM8K
Published: 2024-08-01

Math · 2024-08-01

GSM8K evaluated with the ZeroEval zero-shot prompt format, greedy decoding, and answer parser.

Model	Score
Claude 3.5 Sonnet (Oct 2024)	96.7
GPT-4o (2024-08-06)	96.2
o1-mini	96.1
Llama 3.1 405B Instruct	96.0
Claude 3.5 Sonnet (June 2024)	95.6
Opus 3	95.6
Mistral Large 2 (Instruct 2407)	95.5
GPT-4o (2024-05-13)	95.4
Gemini 1.5 Pro Experimental 0801	95.0
Haiku 3.5	94.5
GPT-4o Mini	94.2
Llama 3.1 70B Instruct	94.2
DeepSeek-V2-Chat-0628	93.9
DeepSeek-Coder-V2-Instruct	93.8
Gemini 1.5 Pro (source unspecified)	93.4
Llama 3 70B Instruct	93.0
Qwen2 72B Instruct	92.7
DeepSeek-V2.5	92.5
DeepSeek-Coder-V2-Instruct-0724	91.5
Claude 3 Sonnet	91.5
Gemini 1.5 Flash (source unspecified)	91.4
Gemma 2 27B IT	90.2
Claude 3 Haiku	88.8
Gemma 2 9B IT	87.4
Reka Core 20240501 (ZeroEval label)	87.4
Athene-70B	86.7
Yi 1.5 34B Chat	84.1
Llama 3.1 8B Instruct	84.0
Mistral NeMo Instruct 2407	82.8
Yi-Large	82.6
Phi-3.5 Mini Instruct	82.0
GPT-3.5 Turbo 0125	80.4
C4AI Command R+	80.1
Qwen2 7B Instruct	80.1
Llama 3 8B Instruct	78.5
Yi 1.5 9B Chat	76.4
Phi-3-mini-4k-instruct	75.5
Reka Flash (2024-02-26)	74.7
Mixtral 8x7B Instruct	70.1
Command R	53.0