← All benchmarks

τ^τ-bench release v1 (hyper-tau-bench)

Agents · 2026-09-04

τ^τ-bench (pronounced hyper-tau-bench; engineering name Hyper-τ), from Sierra's τ-bench team, makes building a customer-service agent the task. A developer configuration, a coding harness such as Claude Code, Codex, Kimi Code or OpenCode driving a builder model, works offline in a sandboxed container with an eight-hour build budget: it recovers the business's policies from its records (documents, support transcripts, call recordings, screenshots and flowcharts, about 1,700 files for banking), may interview a simulated client who alone holds some requirements, works through a client REST API that can be subtly defective, and submits an executable agent that must serve from a fixed menu of models within a per-conversation credit budget. That agent is then deployed against held-out τ³-bench-style simulated users and graded on verifiable outcomes; a task scores its mean reward less any budget-overage penalty, floored at zero. The score is the unweighted mean over all 53 release-v1 tasks (6 airline_plus, 6 retail_plus, 6 telecom and 35 banking_knowledge, so banking dominates), from one construction trial per task. It is not a duplicate of τ-bench, τ²-bench, τ³-bench or Δτ-Bench, which score a model acting as the customer-service agent: here the model builds the agent, and τ³-bench supplies only the inner evaluation loop.

Benchmark source: arXiv

Top models (higher is better)