Omni-MATH (HELM Capabilities)

Name: Omni-MATH (HELM Capabilities)
Published: 2025-03-20

Math · 2025-03-20

HELM Capabilities' Omni-MATH run: 1,000 olympiad-level problems from Omni-MATH, solved zero-shot, with each final answer checked for equivalence with the reference by an ensemble of GPT-4o (2024-05-13) and Llama 3.1 405B Instruct Turbo, joined by Claude 3.5 Sonnet (2024-10-22) for runs first published through release v1.14.0; Anthropic deprecated Claude 3.5 Sonnet before v1.15.0. An item scores the share of judges that accept it, and the benchmark score is the mean over items.

Top models (higher is better)

Model	Score
GPT-5 Mini	72.2
o4-mini	72.0
Qwen3 235B A22B Instruct 2507	71.8
gpt-oss-120b	68.8
Kimi K2 Instruct	65.4
GPT-5	64.7
Opus 4	61.6
Sonnet 4	60.2
gpt-oss-20b	56.5
Haiku 4.5	56.1
Sonnet 4.5	55.3
Qwen3-235B-A22B	54.8
GPT-5 Nano	54.6
Gemini 2.5 Flash-Lite	48.0
Qwen3-Next-80B-A3B-Thinking	46.7
GPT-5.1	46.4
Gemini 2.0 Flash	45.9
DeepSeek-R1-0528	42.4
Llama 4 Maverick Instruct FP8	42.2
Gemini 2.5 Pro Preview 03-25	41.6
Palmyra X5	41.5
DeepSeek-V3	40.3
GLM 4.5 Air FP8	39.1
Gemini 2.5 Flash Preview 04-17	38.5
Gemini 2.0 Flash-Lite Preview 02-05	37.4
Llama 4 Scout Instruct	37.3
Gemini 1.5 Pro 002	36.4
Nova Premier	35.0
Claude 3.7 Sonnet	33.0
Qwen2.5 72B Instruct	33.0
Gemini 1.5 Flash 002	30.4
Granite 4.0 H Small	29.6
Qwen2.5 7B Instruct	29.4
GPT-4o (2024-11-20)	29.3
Mistral Large 2.1 (Instruct 2411)	28.1
GPT-4o Mini	28.0
Claude 3.5 Sonnet (Oct 2024)	27.6
Llama 3.1 405B Instruct	24.9
Mistral Small 3.1 24B Instruct 2503	24.8
Nova Pro	24.2