PersonaCUABench: Benchmarking Computer-Use Agents Serving Diverse Personas
Haoming Li, Defu Cao, Xu Hu, Yingtao Luo, Yiyang Wang, Xianghe Xu, Jiashu He, Wanting Jiang, Yifan Zhang, Yunhan Zhao, Siyang Wu, Chen Yang, Haixin Wang, Zheyuan Deng, Taoran Li, Jingwen Zhang, Zhejian Peng, Shushu Liang, Yue Zhao, Bingzhe Li, Feng Chen, Yifan Wang, Jintao Huang, Yijun Wang, Yuexing Hao, Xiaomin Li
cs.HC, cs.AI, cs.CY, cs.LG
2026-10-16
PersonaCUABench holds 74 tasks fixed and varies only who asks: nine computer-use agents drop from 0.689 to 0.441 completion when one of 44 personas rephrases the request, and most of the loss comes after the agent has heard the requirement.
Computer-use agents (CUAs) are benchmarked against fixed instructions: a complete spec is handed over, then the end state is checked or a rubric is applied. Real users do not talk like a spec. They omit requirements, add personal constraints such as asking to see a draft before anything is sent, and describe goals in layman's terms. An agent can pass every check and still fail the person who asked. Earlier personalization work hands the agent a stored profile or browsing history; this paper does the opposite, giving the agent nothing but the dialogue. The question becomes whether the agent serves each person, or only the task.
A three-stage controlled experiment where the task stays fixed and only who asks changes.
Each backbone runs 370 cells (74 tasks × 4 personas + the original request); nine backbones give 3,330 runs.
| Condition | Task completion | Satisfaction at delivery |
| Original request | 0.689 | 4.04 |
| expert | 0.504 | 3.59 |
| practical | 0.445 | 3.33 |
| young | 0.414 | 3.42 |
| senior | 0.402 | 3.10 |
All nine backbones drop, with relative losses from 28% (Grok 4.6) to 45% (MiniMax M3) and every 95% interval below zero. Seniors lose the most (−42%), experts the least (−27%).
The 1,230 loss cells break down as: 59.9% are requirements the agent engaged with and still got wrong; 17.1% were never raised by the user; 11.2% are stops the user requested (a draft, a review) that the original checker scores as failures; 9.7% are stops the agent made on its own.
Completion and satisfaction diverge in both directions. Of 646 fully successful persona cells, only 62.7% get the top satisfaction rating, against 84.1% under the original request; of 800 failed action cells, 56% are judged done at delivery. Misreporting is one cause: final replies contradict the outcome in 45% of information loss cells, and misreporting tracks higher satisfaction (3.47 vs 3.13).
Two interventions on GPT-6 Astra fail to close the gap. Handing the agent the user's profile lowers information-task completion by 0.027. Forcing at least one question moves persona completion from 0.434 to 0.421, and drops original-request action tasks from 0.583 to 0.472. Only 24% of the 609 questions in the main experiment target a requirement the user had omitted.
The benchmark measures a gap that fixed-instruction leaderboards cannot see: the same agents lose about a third of their completion when a person, not a spec, makes the request. The diagnosis is more useful than the headline. Most loss happens after the agent has heard the requirement, so the bottleneck is carrying user constraints through a long action sequence, not extracting them; both intuitive fixes, more context and more questions, were tested and failed, a directly usable negative result for agent builders. Eval designers should also note that who asks moves scores as much as which model runs (expert personas span 0.14 across models, seniors 0.16), and Gemini 3.8 Flash goes from near the top on original requests (0.734) to seventh on senior requests (0.360).
Author-stated scope: web information and action tasks only, no desktop apps, chat, or coding; one request per episode with no persistent user memory; persona construction may encode stereotypes and simulated satisfaction cannot replace real users; interventions ran only on GPT-6 Astra.
The appendices add reasons for caution. The persona agent, the satisfaction rater, and every judge are the same model, GPT-5.6-terra, and the rater is rebuilt holding the original instruction, information a real user would not have. Its exact agreement with human annotators on satisfaction is around 37% (κ 0.37-0.44); an external judge matches humans better (κ 0.52-0.60). Agreement on the raised and engaged labels is weak (κ 0.20-0.55). The 36% headline also mixes two things: genuine agent failure, and checkers counting user-requested stops as failures. The paper itself cites work showing LLM-simulated users are more verbose and polite than real ones, so the drop with real users is unverified.