Multi-IF

Name: Multi-IF
Published: 2024-10-21

Chat & Writing · 2024-10-21

Multi-IF extends IFEval-style verifiable instruction following to 4,501 three-turn conversations in eight languages, scoring how many of the accumulated instructions each turn satisfies. The headline figure averages accuracy over turns and languages.

Top models (higher is better)

Model	Score
LFM3-1.7B	89.1
LFM2.5-2.6B	84.8
Qwen3.5-9B	84.4
LFM2.5-24B-A2B	84.4
Nemotron 3.5 Lightning 30B A3B (precision unspecified)	82.3
Gemma 4 26B A4B IT	82.2
Qwen3.6 35B A3B	81.5
Qwen3 235B A22B Thinking 2507	80.6
Qwen3.5-4B	80.3
o3	80.3
o3-mini	79.5
Qwen3 VL 235B A22B Thinking	79.1
O4 Mini	78.0
Qwen3 VL 32B Thinking	78.0
o1 (2024-12-17)	77.9
Gemini 2.5 Pro	77.8
Qwen3-Next-80B-A3B-Thinking	77.8
Qwen3 235B A22B Instruct 2507	77.5
Qwen3 4B Thinking 2507	77.3
Qwen3 30B A3B Thinking 2507	76.4
Qwen3-VL-235B-A22B-Instruct	76.3
Kimi K2 Instruct	76.2
Qwen3 Next 80B A3B Instruct	75.8
Llama 4 Maverick Instruct	75.5
Qwen3 VL 8B Thinking	75.1
Qwen3 14B	74.8
Gemini 2.5 Flash	74.4
Qwen3 VL 4B Thinking	73.6
Qwen3 32B	73.0
Qwen3 VL 30B A3B Thinking	73.0
Qwen3 Omni 30B A3B Thinking	72.9
Qwen3-30B-A3B	72.2
Qwen3 VL 32B Instruct	72.0
Qwen3-235B-A22B	71.9
Qwen3 8B	71.2
GPT-4.1	70.8
GPT-4.5	70.8
ChatGPT-4o Latest observed 2025-03-27	70.4
Gemma 3 27B IT	69.8
Qwen3 4B Instruct 2507	69.0