Mind the Sim2Real Gap in User Simulation for Agentic Tasks
Xuhui Zhou, Weiwei Sun, Qianou Ma, Yiqing Xie, Jiarui Liu, Weihua Du, Sean Welleck, Yiming Yang, Graham Neubig, Sherry Tongshuang Wu, Maarten Sap
COLM 2026
cs.AI
2026-03-12
451 humans vs 31 LLM user simulators on τ-bench: models over-cooperate, lifting agent success from 63.6% to 77.8%. Best USI 76.0, humans 92.7.
Agent evals such as τ-bench now use an LLM as the customer. The model writes user turns and also scores the conversation. The same pattern shows up in support, coding, clinical, and social settings. Faithfulness to real users is usually assumed.
Carnegie Mellon's Language Technologies Institute treats that assumption as a Sim2Real gap, the robotics term for policies that work in simulation and fail in the world. Three questions follow: do simulators talk like people, do they grade like people, and can τ-bench's 0/1 database-state reward replace human feedback. They ran the full protocol with 451 Prolific annotators on 165 airline and retail tasks, kept the agent fixed at GPT-5.2, and compared 31 simulators against those traces.
τ-bench is the case study because the user, the tool-using agent, and the automatic reward sit in one loop. Swap the LLM user for a person and the rest of the stack stays put. Tasks cover policy-constrained support work such as rebooking, returns, and order lookup, with enough turns for behavioral stats.
Behavior is scored on four axes. Communication style (D1) covers politeness, length, and punctuation. Information pattern (D2) counts how many identifiers, emails, order IDs, land in each turn. Clarification (D3) tracks hedging versus certainty. Error reaction (D4) tracks emotion, blame, and strategy switches. Lexical cues come from LIWC2015 and the NRC emotion lexicon; structural cues from regex. Each metric is aligned to the human population with a Sorensen-Dice coefficient, 100 meaning a perfect match on that rate.
Outcomes get two more scores. Expected Calibration Error (ECE) bins tasks by difficulty and compares simulator versus human success of the same agent. After each dialogue, both humans and simulators fill the same survey: a five-way task-success judgment plus efficiency, question load, answer effort, human-likeness, flow, and reuse. Mean absolute error on those ratings becomes Eval. The User-Sim Index (USI) averages the four behavior scores, (1-ECE)100, and Eval. Models without surveys drop Eval and average five terms. Human-human agreement across three annotator batches is the ceiling. The 31 simulators split into 18 proprietary, 9 open-source, and four specialized user models: CoSER-8B, UserLM-8B, HumanLike-7B, HumanLM-opinion.
DeepSeek-V3.1 leads the simulators at USI 76.0. Humans sit at 92.7. Best proprietary model is Gemini-2.0-Flash at 74.7. None of the four specialized models close the gap; HumanLM-opinion lands at 46.9. Claude-3-Haiku is the closest rater (Eval 78.3) but its communication-style score is 22.1, so USI falls to 61.8. A good judge is not a good customer.
| Simulator | USI | Eval |
| Human (inter-annotator) | 92.7 | 97.4 |
| DeepSeek-V3.1 | 76.0 | 74.3 |
| Gemini-2.0-Flash | 74.7 | 73.7 |
| Llama-4-Maverick | 73.7 | 76.7 |
| GPT-5.1 | 73.5 | 72.1 |
| Claude-3-Haiku | 61.8 | 78.3 |
| HumanLM-opinion | 46.9 | 61.6 |
All four behavior axes miss. On D1, 1.0% of GPT-4o turns are short versus 29.0% for humans, and 49.0% are polite versus 15.3%. On D2, UserLM-8B dumps 4.8 identifier-like tokens per turn versus 2.6, often reading out name, email, and order ID in the first message. On D3, GPT-4o hedges on 14.6% of turns versus 7.3% for humans, even when the instruction is unambiguous; UserLM-8B almost never hedges (3.0%) and overuses certainty markers (10.6% versus 1.0%). On D4, people push back ("This is ridiculous, why can't you waive the fees"); simulators politely switch tactics (GPT-4o 19.1% pivots, CoSER 16.5%, humans 8.4%). UserLM-8B sometimes opens as the support agent.
That cooperative user turns the benchmark into easy mode. The same GPT-5.2 agent succeeds 63.6% of the time with humans, 77.8% with MiniMax-M2.5, 76.6% with Claude-3.5-Sonnet, 76.2% with Kimi-K2.5. Specialized simulators fall below the human baseline (CoSER-8B 57.6%, HumanLike-7B 42.6%), which the paper attributes to weak instruction following on a hard role-play, consistent with their low USI.
Judging is skewed too. GPT-5.1 overrates human-likeness by 55% of the scale and overall score by 18%, with mean deltas of +1.11 on human-likeness and +0.83 on reuse, while task completion is 0.15 lower. Experience ratings run high; success ratings run slightly low.
τ-bench's binary reward is largely orthogonal to what people report. Among reward=0 dialogues, 70.6% are still judged successful by humans. Among reward=1, 33% are judged failed or only partial. Score distributions for the two reward groups barely separate on overall quality, efficiency, or human-likeness.
Higher general capability does not reliably mean a better user. Outside the GPT family, Chatbot Arena Elo does not track USI. A persona prompt for GPT-5-mini, written from these findings and tested on 50 tasks, moved some surface features toward humans while USI fell from 70.9 to 64.6 and ECE worsened from 0.18 to 0.29. Politeness and front-loading improved; uncertainty and dash use overshot; emotion stayed flat; pivoting got worse.
Anyone reporting τ-bench-style numbers with an LLM user is measuring against a partner that hands over every identifier and rarely complains. Part of the success rate is a gift from the simulator.
Simulators remain useful for cheap sweeps. The practical protocol in the paper is: compare widely with simulators, then run a smaller human set that covers each domain and both easy and hard tasks, and check whether rankings still hold. Report behavior, task outcomes, and user ratings separately. Prompting a more human persona did not close the gap in the one ablation they ran.
This is an empirical warning, not a new training recipe. The action is narrow: do not write simulator scores as evidence about real users.
The authors flag several. D1-D4 rest on lexicons and regex; D4 counts emotion and pivots over all user turns, not only turns after a verified agent error, and they do not report how often that association is correct. Annotators were role-playing, not customers with a real booking to change, and the pool is mostly English-speaking US adults. The main study fixes one agent; a Gemini-3.1-Pro rerun preserves ranking more than absolute scores. Known τ-bench task and reward issues can move absolute numbers; the human-simulator contrast still holds because tasks are matched. Coding, browsing, clinical, and open social settings were not tested.
Two more gaps sit in the writeup. The claim that specialized models fail from weak instruction following is not backed by a trajectory breakdown: it is unclear whether they cannot play the customer or whether they play it so oddly that the agent collapses. USI averages six axes with equal weight. D2 scores cluster high (often 70-88) while D1 and D4 spread out, so a single index mixes "talks unlike a person" with "dumps too many IDs". The paper already says USI is a summary, not a substitute for the parts.