$τ$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains
Shunyu Yao, Noah Shinn, Pedram Razavi, Karthik Narasimhan
cs.AI, cs.CL
2024-06-18
Sierra's τ-bench tests service agents on users, APIs, and policy. gpt-4o hits 61.2% pass^1 on retail, 35.2% on airline; pass^8 on retail is under 25%.
Most agent benchmarks hand the model a complete instruction and let it talk to a website or an API on its own. Live customer service does not work that way. Users drip information across turns, domain rules live in a policy document, and the same request phrased differently should still end in the same database state. τ-bench (Tool-Agent-User Interaction Benchmark), from Sierra, scores all three at once: multi-turn talk with a simulated user, domain APIs, and policy compliance.
The first two domains are retail returns and airline rebooking. Schemas and rules can be written down, databases can be synthesized, and each task can be constrained so that policy plus user preference admit only one legal write.
Each task is a partially observable decision process. The agent never sees the database and can only read or write through tools. A gpt-4-0613 user simulator sees the chat, not the tool trace. Domain policy sits in the system prompt (basic economy cannot be changed after 24 hours, and so on). The hidden user instruction encodes identity, intent, and preferences tightly enough that only one compliant database outcome remains.
Reward is binary: 1 only if the final database matches the annotated goal and the agent's messages contain required numbers. Conversations may wander; the write must not. The authors note that a return issued without explicit confirmation can still score 1, so the rule is necessary and not sufficient.
They also define pass^k: the probability that all k i.i.d. trials of the same task succeed. That is the opposite of pass@k used in code generation, where one success among k tries counts. Customer service cares about every trial, not a lucky one.
Construction is three stages: humans design schemas, APIs, and policies; gpt-4 expands database rows; humans write user instructions and iterate them against a gpt-4-turbo function-calling agent until the outcome is unique. Retail has 115 tasks (500 users, 50 products, 1,000 orders, 7 write APIs, 8 read APIs). Airline has 50 (300 flights, 2,000 reservations, 6 write, 7 read). Cap is 30 agent steps. Agent temperature is 0, user temperature is 1. Main numbers average at least three trials per task.
Native function calling beats text-format ReAct on every strong model. gpt-4o leads: 61.2% pass^1 on retail, 35.2% on airline, 48.2% domain-weighted. gpt-4-turbo scores 57.7/32.4, claude-3-opus 44.2/34.7, gpt-3.5-turbo 20.0/10.8. Llama-3-70B, run with text ReAct, sits at 14.8/14.4. Open-weight models lag. Airline is clearly harder.
Consistency is worse. On retail, gpt-4o's average success is above 60%, yet pass^8 falls below 25%. Same policy, same goal, a different user phrasing, and the run flips.
Dropping the policy from the system prompt moves gpt-4o from 61.2% to 56.8% on retail and from 33.2% to 10.8% on airline. Retail rules are close to common sense, so the model was barely using the handbook. Airline baggage and cabin rules are ad hoc; remove the document and the score collapses. gpt-3.5-turbo barely moves (20.0 to 14.5, 10.8 to 9.6). It cannot digest the complex rules in the first place.
A manual audit of 115 gpt-4o retail traces (one trial per task, 40 failures, 65.2% pass^1, a different protocol from the 61.2% three-trial table) found 4 annotation bugs and 36 agent errors: wrong arguments 33.3%, wrong or missing info 22.2%, wrong decision type 25.0%, partial handling of compound requests 19.4%. Hallucinated IDs per retail task: 0.46 for gpt-4o, 2.08 for gpt-3.5-turbo function calling, 6.34 for Act-only. Cost for gpt-4o plus a gpt-4 user on retail is about $0.38 + $0.23 per task, roughly $200 for one pass over 115 tasks, almost all of it the long system prompt.
| Model | Retail pass^1 | Airline pass^1 | Weighted avg |
| gpt-4o | 61.2 | 35.2 | 48.2 |
| gpt-4-turbo | 57.7 | 32.4 | 45.1 |
| claude-3-opus | 44.2 | 34.7 | 39.5 |
| gpt-3.5-turbo | 20.0 | 10.8 | 15.4 |
| Llama-3-70B | 14.8 | 14.4 | 14.6 |
This is one of the closer public proxies for production customer-service agents: a user in the loop, a policy in the loop, and a database-state score instead of an LLM judge. pass^k separates "works on average" from "works every time", which is what an SLA actually asks. The code and data are public, and later work has treated τ-bench as a standard agent yardstick.
For anyone shipping a bot, 61% once and under 25% across eight trials means function calling alone will not carry a million conversations. The failure modes are specific: inventory reasoning, write APIs that may be called only once, compound requests that get half-done. Those are better levers than "try a bigger model".
The user is an LM. It miscalculates, forgets, and follows the agent's suggestion. The paper itself notes that users do not know an exchange tool can be called only once per order, and will approve a one-item swap. Task instructions were tuned against gpt-4-turbo function calling, which biases the set toward that family. Reward ignores missing confirmation, so policy following is only partly scored. There are two customer-service domains; medical, tax, and legal work is left as future. The main airline number for gpt-4o is 35.2%; the no-policy ablation writes 33.2%. Trial counts are not fully aligned.