On Evaluating User Simulators for Interactive Agents
Hengguang Zhou, I-Hung Hsu, Li Chen, Priyanshi Shrivastava, Anastasios Angelopoulos, Wei-Lin Chiang, Ion Stoica, Cho-Jui Hsieh
cs.LG
2026-11-19
Injecting 10 failure modes into simulated users shows prior benchmarks catch only 5-6, some even favoring users that accept broken work; a new three-level suite catches all 10.
LLM agents increasingly do multi-turn work for users (code, games, design assets), and honest evaluation of those interactions needs real people: someone states requirements, inspects the output, pushes back. Recruiting them for every eval run is expensive and hard to repeat, so teams substitute user simulators, an LLM playing the user. Whether a simulator is any good has been checked with metrics from prior work (SimulatorArena, VISTA, SWE-Together) that measure similarity to real users or consistency with user specifications. Nobody had tested whether those metrics catch the failures that make a simulator unfit. A simulated user can sound perfectly natural while approving broken work, and earlier work already showed simulator-based agent scores drift from human-based ones. This paper points the finger at the metrics themselves.
The core move is perturbation testing: treat the evaluation metrics as the object under test, a meta-evaluation. Failure modes come from an analysis of 442,516 real sessions and 20,100 replays plus failures reported in the literature, condensed into 10 perturbations injected into the simulator's setup (all replays use gpt-5.6-terra as the simulated user). Each source session and agent runs twice, once control and once perturbed; a metric passes if perturbed replays score reliably lower (sensitivity), and separately the authors test whether scores separate controls from perturbed candidates when used as a filter (AUROC).
The 10 perturbations map to three levels: downstream utility (agent favoritism toward a selected agent; a profile-extraction artifact that freezes dissatisfaction with the original agent into a persistent user trait), population fidelity (approving broken work, premature stopping, front-loading requirements into the opener), and trace fidelity (swapping the goal, removing constraints, swapping persona, co-debugging for the agent, switching language mid-session).
The proposed suite measures the same three levels: L1 ranks 9 agents by simulated feedback and compares against a reference ranking derived from real-user reactions via Kendall's τ; L2 compares four families of population statistics between simulated and real users (dialogue-act distribution, interaction dynamics, reaction-to-error patterns, user-vs-agent share of task information); L3 checks each replay for goal drift, constraint preservation, and persona consistency. The headline score is the geometric mean.
| Suite | Perturbations detected (of 10) |
| SimulatorArena | 6 |
| VISTA | 5 |
| SWE-Together | 6 |
| This suite | 10 |
All three external suites miss agent favoritism even though it shifts agents' scores and ranks. Composite scores even reward the accept-broken-work population: VISTA's knowledge consistency drops to an AUROC of 0.01 at full replacement, below chance. On filtering, L1 reaches AUROC 0.99 at full favoritism replacement versus 0.64 for the best external metric, and retains 98% of control populations at a threshold admitting 5% of perturbed samples, versus 10% for Sim2Real. The trace-level persona score reaches 0.90 AUROC with real-user traces as positives and retains 68% of them, versus 28% for the best external benchmark.
Six simulators evaluated on the same 351 real sessions (348 users, 23 languages; artifacts include web apps, games, design assets):
| Simulator | L1 | L2 | L3 | Overall |
| Sonnet 5 config | 47 | 87 | 83 | 70 |
| SimulatorArena (gpt-5-mini) | 57 | 79 | 74 | 69 |
| Terra config | 42 | 73 | 84 | 64 |
| Luna config | 40 | 72 | 78 | 61 |
| τ2-bench (gpt-4.1) | 33 | 82 | 74 | 59 |
| MIMESIS-9B | 27 | 82 | 61 | 51 |
No simulator leads on all three levels: SimulatorArena is best at ranking agreement (Kendall's τ 0.67 vs 0.44 for the Sonnet 5 config), Sonnet 5 best on population fidelity, Terra best on trace fidelity.
If you evaluate or train agents against simulated users, the takeaway is direct: a high score on existing criteria is not evidence the simulator is usable. When interactions revolve around artifacts, natural dialogue and grounded acceptance are different things, and population-level statistics miss a single replay pursuing the wrong goal or dropping a constraint. The three-level diagnostic lifts straight into an internal eval pipeline, and the 351 real multi-turn sessions are released. Teams building RL training environments face the same issue, since simulator distortion flows into the training signal. The qualitative examples are vivid: a real user only reports 'the color isn't updating,' while the simulated user diagnoses Profile.isanimated as a property and prescribes the fix, with Terra producing code-bearing turns at 3.46× the real-user rate; in another replay the simulator waives the 8K resolution requirement and approves a 1920×1080 file.
The authors concede that sensitivity does not guarantee useful filtering: with control replays as positives, the persona family manages only 0.62 AUROC, and a strict threshold admitting 5% of perturbed samples keeps just 10% of controls. Premature stop exposes a blind spot in their own framework: the control simulator produces more turns than real users, so truncation moves session statistics closer to the real reference, all four L2 families miss it, and only a trace-level constraint metric catches it, weakly (win rate 0.56). Two further concerns: the pipeline leans on Claude Sonnet 4.6 as judge for reactions, drift, and constraints, and that judge's reliability is not itself perturbation-tested; the L1 reference ranking is an external input, so a different reference could reshuffle the six simulators. The perturbations are also injected and carry stronger signal than the subtler degradation real simulators show.