τ³-Banking (Mercor run, bm25_grep retrieval)
Agents
Mercor's run of τ³-Banking, the 97-task banking_knowledge domain of tau2-bench v1.0.1, in which a support agent searches about 700 bank policy documents and uses tools to resolve a simulated customer's issue, graded on the final database state and the agent's actions. Mercor gives each agent the offline bm25_grep retrieval profile in Tau's half-duplex agent (200 steps, 10 tool errors, 3 hours), simulates the user with GPT-5.2 at low reasoning effort, has Gemini 3.5 Flash judge the one natural-language assertion, and averages pass@1 over three runs. Kept as a protocol component beside the all-tools and BM25 runs in the banking family.
Top models (higher is better)
| Model | Score |
|---|---|
| Kimi K3.1 Pro | 61.5 |
| Gemini 4 Argon | 58.1 |
| Gemini 4 Carbon | 55.7 |
| Claude Opus 5.5 | 55.0 |
| Claude Fable 5.1 | 54.0 |
| Claude Sonnet 5.5 | 54.0 |
| Muse Spark 1.3 | 52.2 |
| Muse Wonder | 51.9 |
| Gemini 3.8 Flash | 48.8 |
| GPT-6 Astra | 46.7 |
| Grok 4.6 | 45.7 |
| Claude Opus 5 | 44.0 |
| Claude Haiku 5.5 | 43.0 |
| GPT-5.6 Sol Pro | 43.0 |
| GPT-6.1 Sol | 43.0 |
| GPT-6 Sol | 43.0 |
| DeepSeek V4.1 Flash 1017 | 41.6 |
| GLM-5.3 | 40.5 |
| Mercury 3 | 40.5 |
| DeepSeek V4 Pro 0813 | 40.2 |
| GPT-6 Luna | 40.2 |
| GPT-5.6 Terra | 38.8 |
| DeepSeek V4.1 Flash | 37.8 |
| GLM-5.3 Flash | 37.8 |
| GPT-6.1 Terra | 37.1 |
| MiniMax M3.1 Flash | 36.1 |
| GPT-5.6 Luna | 35.7 |
| Kimi K3 | 34.7 |
| Beam 1.0 501B A23B | 33.0 |
| Grok 4.7 | 32.6 |
| MiniMax M3 | 21.0 |
| Gemini 3.5 Flash-Lite | 18.6 |
| Nemotron 3 Ultra 550B A55B (NVFP4) | 18.6 |
| Inkling | 15.5 |