τ-bench (Airline)
Agents · 2024-06-17
The airline domain of the original τ-bench (2024): an agent serves an LLM-simulated user over 50 reservation tasks, using airline-booking tools and following a policy document, and a task succeeds only when the final database state matches the goal. Scores report pass@1 (the benchmark's pass^1), the single-trial success rate averaged over trials; the user-simulator model is recorded in score notes. The revised τ²-bench airline domain is a separate benchmark. Higher is better.
Top models (higher is better)
| Model | Score |
|---|---|
| Sonnet 4 | 60.0 |
| Qwen3-Coder-480B-A35B-Instruct | 60.0 |
| Opus 4 | 59.6 |
| Claude 3.7 Sonnet | 58.4 |
| Opus 4.1 | 56.0 |
| o1 (2024-12-17) | 54.2 |
| Gemini 2.5 Flash | 54.0 |
| DeepSeek-R1-0528 | 53.5 |
| Kimi K2 Instruct | 53.5 |
| o3 | 52.0 |
| Gemini 2.5 Pro | 52.0 |
| GPT-4.5 | 50.0 |
| GPT-4.1 | 49.4 |
| gpt-oss-120b | 49.2 |
| O4 Mini | 49.2 |
| Qwen3-Next-80B-A3B-Thinking | 49.0 |
| Qwen3 30B A3B Thinking 2507 | 48.0 |
| Qwen3 4B Thinking 2507 | 48.0 |
| Qwen3 Coder 30B A3B Instruct | 48.0 |
| Qwen3 235B A22B Thinking 2507 | 46.0 |
| Qwen3 235B A22B Instruct 2507 | 44.0 |
| Qwen3 Next 80B A3B Instruct | 44.0 |
| GPT-4o (2024-11-20) | 42.8 |
| gpt-oss-20b | 42.6 |
| Qwen3 30B A3B Instruct 2507 | 40.0 |
| Claude 3.5 Sonnet (June 2024) | 36.0 |
| GPT-4.1 Mini | 36.0 |
| Opus 3 | 34.5 |
| o3-mini | 32.4 |
| Qwen3-235B-A22B | 32.0 |
| Qwen3-30B-A3B | 32.0 |
| Qwen3 4B | 32.0 |
| Qwen3 4B Instruct 2507 | 32.0 |
| Qwen3 32B | 29.0 |
| Haiku 3.5 | 22.8 |
| GPT-4o Mini | 22.0 |
| Claude 3 Haiku | 16.0 |
| GPT-4.1 Nano | 14.0 |