Autonomy Reshapes How Personalization Affects Privacy Concerns and Trust in LLM Agents
Zhiping Zhang, Yi Evie Zhang, Freda Shi, Tianshi Li
COLM 2026
cs.HC, cs.AI, cs.CR
2025-10-06
A 3x3 experiment (N=450): agents that run autonomously but hand back control on detected privacy risk earn more perceived control and trust than step-by-step approval.
For an LLM agent to act on your behalf, it needs your data: Gmail, calendar, notes. The more it reads, the larger the leak surface. Unlike a chatbot, an agent decides on the fly what to send to whom, so one message can carry information you never meant to share into a group chat. Survey studies keep confirming the obvious: perceived risk raises privacy concern and lowers trust and willingness to share.
Industry answers sit at two extremes. Align the model so it learns not to say the wrong thing, or hand the user control over every step. This paper (COLM 2026, Northeastern / UIUC / Waterloo) asks the question both skip: does the agent's autonomy level itself change how users react to personalization? Decision authority allocation, not just model output, as a trust lever.
A 3x3 between-subjects experiment, N=450 (from 538 screened, 50 per cell, US Prolific sample). Participants supplied real personal information in a pre-survey and marked which items they did not want others in the discussion to know. That data, formatted as Notion and calendar JSON, fed a GPT-4o-mini agent that spoke for them in a group chat (a weekly team update or a family trip discussion; the two counterparties were scripted role-play agents).
Two factors, three levels each:
The risk-contingent detector was an LLM-as-a-judge setup on GPT-4o: extract a list of sensitive items from the task context, then judge whether a message contains any. Detection and reminder UI were identical across all nine conditions. Post-task surveys measured privacy concern, trust, and willingness to use; a moderated mediation SEM traced the mechanism through perceived sensitivity, perceived control, and perceived usefulness.
Main effects confirmed the known pattern: full personalization produced the highest privacy concern and lowest trust (concern beta = -2.07 vs no personalization, p<.001).
The autonomy factor is where it gets interesting:
| vs no autonomy | Privacy concern | Trust | Willingness |
| Risk-contingent | beta = -0.79 | beta = +0.49 | beta = +0.67 (n.s.) |
| Full autonomy | n.s. | n.s. | n.s. |
The interaction was significant: full personalization was worst everywhere, but under risk-contingent autonomy the slope across personalization types flattened substantially, attenuating personalization's negative effects.
Mediation analysis isolated perceived control as the only pathway: risk-contingent autonomy raised perceived control by 0.48 (CI [0.09, 0.84]), absorbing the mediation path through which personalization would otherwise hit the outcomes. The sensitivity and usefulness pathways stayed stable across autonomy levels.
One counterintuitive objective measure: 68% of users under risk-contingent autonomy correctly identified privacy leakages in the agent's messages, versus 62% under no autonomy and 58% under full autonomy. Fewer confirmation opportunities, better leakage detection.
For agent builders, this turns "when to ask the user" from intuition into a measured design variable. The finding is specific: confirmation requests should align with risk, not with every step. Step-by-step approval looked safest and delivered neither higher perceived control nor better leak detection.
Escalation on risk is already industry practice (Operator escalates on high-stakes domains, AirGapAgent on out-of-scope requests); this supplies the first controlled experimental backing. The detector comparison adds a deployment note: DeepSeek V4 Flash, a small open-source model, reached precision .896 / recall .958, comparable to GPT-4o, so leak detection can run locally without routing personal data to another cloud.
Theoretically it establishes autonomy alignment as its own object: decision-authority allocation is a property of the human-agent system, not the model alone.
The authors list five. The ones that matter: only two everyday chat scenarios (weekly meeting, family trip), and while scenario random effects were not significant, extrapolation to high-stakes settings (money, health) has no basis here; the whole discussion was simulated, so the felt cost of a real leak is absent; leakage frequency and severity were not deliberately manipulated, so actual exposure varied across participants.
A few more caveats from reading closely: