Atlas

Benchmarks

← All benchmarks

IFEval (prompt-level strict accuracy)

Chat & Writing · 2023-11-14

IFEval's prompt-level strict accuracy: the share of its 541 prompts for which every verifiable instruction is followed under strict checking. It is one of IFEval's four standard metrics and is kept apart from loose or instruction-level averages.

Top models (higher is better)

ModelScore
Qwen2.5-VL-72B-Instruct86.3
Qwen Plus (rolling alias)86.3
Llama 3.1 405B Instruct86.0
Qwen2.5 14B Instruct 1M84.3
Qwen2.5 72B Instruct84.1
Llama 3.1 70B Instruct83.6
Claude 3.5 Sonnet (June 2024)82.6
Qwen2.5 14B Instruct81.0
GPT-4o Mini80.4
Qwen2.5 Instruct 32B79.5
Qwen2 72B Instruct77.6
Llama 3 70B Instruct77.3
Gemma 2 27B IT77.1
Qwen2.5-Turbo76.3
Llama 3.1 8B Instruct75.9
Qwen-Turbo (rolling alias)74.9
DeepSeek-V2.573.9
Qwen2.5 7B Instruct 1M73.0
Llama 3 8B Instruct72.1
Qwen2.5 7B Instruct71.2
Gemma 2 9B IT70.1
Mixtral 8x22B Instruct67.1
Mistral Large 2 (Instruct 2407)64.1
Qwen2-57B-A14B Instruct59.9
Qwen2.5 3B Instruct58.2
Qwen1.5 110B Chat57.5
Qwen1.5 72B Chat55.8
Qwen2 7B Instruct54.7
Phi-3.5 Mini Instruct52.1
Gemma 2 2B IT51.0
Qwen1.5 32B Chat50.3
Qwen2.5 1.5B Instruct42.5
Qwen1.5 7B Chat38.3
Qwen2 1.5B Instruct29.0
Qwen2.5 0.5B Instruct27.9
Qwen1.5 1.8B Chat16.8
Qwen1.5 0.5B Chat14.6
Loading Atlas data…