Berliner study: existing user-simulator metrics catch only 5-6 of 10 injected failures
geoffwolfe · x · 2026-11-22
UCLA and Berkeley researchers (incl. Ion Stoica, Cho-Jui Hsieh) released "On Evaluating User Simulators for Interactive Agents," a meta-evaluation of the metrics used to judge user simulators.
- Motivation: agents are increasingly evaluated against simulated users, but simulator quality is poorly measured. A real replay shows the risk: a real user insisted on 8K images; the simulated replacement accepted 1080p, letting the agent "finish" substandard work.
- Method: the authors deliberately perturb simulated replays with 10 failure types (altered goals, omitted constraints, premature stopping, accepting broken work) and test whether metrics penalize them.
- Findings: evaluation methods from three prior studies detect only 5-6 of 10 perturbations; combined scores even favor populations instructed to accept broken work over unmodified controls.
- New suite: combines utility for evaluating agents with population-level and trace-level fidelity to real users; it detects all 10 perturbations under the same criterion.
- Evaluated six user simulators across 351 real multi-turn sessions from 348 distinct users.
More from coding & agent
- MCP lead maintainer solicits feedback while also working on Claude Code — dsp_ · 2026-11-22
- DeepORG open-sources an AI agent that runs across your org's databases, APIs and code — dosco · 2026-11-22
- If an agent can still move money after its prompt says no, the boundary is in the wrong place — ExecLayer_io · 2026-11-22
- AutoRAGBench: LLM agents build RAG systems beating the best published human designs — omarsar0 · 2026-11-22
- Stop @-ing your agent in Slack — let Jev decide when it should respond — DavidWells · 2026-11-22
- Intent adds notch launcher, Claude Code and Codex session import — Wattenberger · 2026-11-22