τ^τ-bench release v1 (banking_knowledge)
Agents · 2026-09-04
The banking_knowledge block of τ^τ-bench release v1, Sierra's agent-construction benchmark in which a coding harness driving a builder model must build a customer-service agent from a business's records, a simulated client and a client API, and is scored by deploying that agent against held-out τ³-bench-style simulated users (a task scores the agent's mean reward less any budget-overage penalty, floored at zero). Its 35 tasks are a retail banking domain (cards, deposits, disputes and transfers, with six embedded-policy journey subdomains) whose records hold 2,969 atomic policy facts, against 85 to 155 in each other domain, so it carries most of the gap to the reference ceiling. The score is the mean over the 35 tasks from one construction trial each; with the other three domain blocks it makes up the 53-task overall score (hyper-tau-bench-v1). Like that score, it rates the model as the builder of the agent, not as the customer-service agent that τ-bench, τ²-bench and τ³-bench score.
Benchmark source: arXiv