τ-bench (Retail)
Agents · 2024-06-17
The retail domain of the original τ-bench (2024): an agent serves an LLM-simulated user over 115 customer-service tasks, using order-management tools and following a policy document, and a task succeeds only when the final database state matches the goal. Scores report pass@1 (the benchmark's pass^1), the single-trial success rate averaged over trials; the user-simulator model is recorded in score notes. The revised τ²-bench retail domain is a separate benchmark. Higher is better.
Top models (higher is better)
| Model | Score |
|---|---|
| Opus 4.1 | 82.4 |
| Opus 4 | 81.4 |
| Claude 3.7 Sonnet | 81.2 |
| Sonnet 4 | 80.5 |
| Qwen3-Coder-480B-A35B-Instruct | 77.5 |
| Gemini 2.5 Pro | 74.8 |
| o3 | 73.9 |
| o1 (2024-12-17) | 73.5 |
| O4 Mini | 71.8 |
| Qwen3 235B A22B Instruct 2507 | 71.3 |
| Kimi K2 Instruct | 70.7 |
| Qwen3-Next-80B-A3B-Thinking | 69.6 |
| Qwen3 Coder 30B A3B Instruct | 68.7 |
| GPT-4.5 | 68.4 |
| GPT-4.1 | 68.0 |
| gpt-oss-120b | 67.8 |
| Qwen3 235B A22B Thinking 2507 | 67.8 |
| Qwen3 30B A3B Thinking 2507 | 67.8 |
| Qwen3 4B Thinking 2507 | 66.1 |
| Qwen3-235B-A22B | 65.2 |
| Gemini 2.5 Flash | 65.2 |
| DeepSeek-R1-0528 | 63.9 |
| Claude 3.5 Sonnet (June 2024) | 62.6 |
| Qwen3-30B-A3B | 61.7 |
| Qwen3 Next 80B A3B Instruct | 60.9 |
| GPT-4o (2024-11-20) | 60.3 |
| Qwen3 30B A3B Instruct 2507 | 59.1 |
| o3-mini | 57.6 |
| GPT-4.1 Mini | 55.8 |
| gpt-oss-20b | 54.8 |
| Qwen3 32B | 52.8 |
| Haiku 3.5 | 51.0 |
| Qwen3 4B Instruct 2507 | 48.7 |
| Opus 3 | 45.1 |
| GPT-4o Mini | 44.0 |
| Qwen3 4B | 33.9 |
| Claude 3 Haiku | 18.2 |