IFEval (instruction-level strict accuracy)

Name: IFEval (instruction-level strict accuracy)
Published: 2023-11-14

Chat & Writing · 2023-11-14

IFEval scored at the instruction level with strict checking: the share of all individual verifiable instructions across the 541 IFEval prompts that the response satisfies exactly, rather than the share of prompts whose every instruction is satisfied. Release reports often give this alongside or instead of prompt-level strict accuracy, and it runs several points higher, so it is kept as a separate measurement. Higher is better.

Top models (higher is better)

Model	Score
Llama 3.3 70B Instruct	92.1
Nemotron 3 Nano 4B	92.0
Qwen3 14B	91.3
Nemotron Nano 9B V2	90.3
Llama 3.1 405B Instruct	89.2
DeepSeek-R1	88.8
Llama 3.3 Nemotron Super 49B V1.5	88.6
QwQ-32B	86.3
Nemotron-4 340B Instruct	86.1
DeepSeek-R1-Distill-Llama-70B	85.1
Nemotron Nano 12B V2 VL BF16	84.7
Qwen2 72B Instruct	84.2
Llama 3.1 Nemotron Nano 4B V1.1	82.6
Llama 3.1 Nemotron Nano 8B V1	82.1
Llama 3.1 8B Instruct	81.8
Nemotron-H 8B Reasoning 128K	79.6
DeepSeek R1 Distill Llama 8B	73.4