Anthropic Study Finds Analysts Follow AI Agents' Judgment Without Noticing

Anthropic's alignment team (Amaç Herdağdelen, Hailey Joren, Alex Tamkin) released a randomized controlled experiment, "How AI Agents Shape Expert Judgment," one of the first studies to systematically measure agents' influence on human professional judgment. The experiment recruited 50 professional data analysts with causal inference experience and had them use Claude Code for statistical analysis for a fictional agricultural cooperative. The tasks embedded 4 key decision points where "both options were reasonable," testing whether experts still make critical decisions after delegating work to an agent.

Confirmed

• Analysts followed the agent's leaning in 86% of decisions, versus a theoretical baseline of 50% if there were no influence.
• The agent was explicitly instructed to make the most honest case for its preferred option, to change its position if the evidence warranted, and never to mislead the analyst; compliance remained high regardless.
• When asked afterward to justify each decision, 77% of analysts repeated the agent's arguments in at least one response, sometimes verbatim; only 15% explicitly credited the agent; about half felt the final recommendation was "more their own."
• The study deliberately amplified the agent's leanings to measure their effect; Alex Tamkin cautioned that real-world effects in everyday work may differ.

Why it matters

• The study reveals a kind of "unaware compliance": experts subjectively feel they retain decision-making authority while objectively adopting the agent's position at high rates. For industries that rely on expert judgment (healthcare, law, financial analysis), this suggests human oversight may be hollowed out once work is delegated to AI.
• Tamkin accordingly argues that agents' influence on human decisions must be addressed at the system design level, and that academia needs a deeper understanding of how agents shape people's decisions at work and in life.

2026-10-28 ~ 2026-10-28 · 12 related posts

• [source] Anthropic RCT: Analysts Follow AI Agent's Lean on 86% of Judgment Calls, Mostly Unaware — AlexTamkin · 2026-10-28
• Anthropic RCT with 50 analysts tests whether experts still call the shots when delegating to agents — AlexTamkin · 2026-10-28
• Even Told to Be Honest and Persuadable, the Agent Swayed 86% of Analyst Decisions — AlexTamkin · 2026-10-28
• [source] 77% of Analysts Parroted the Agent's Arguments, Only 15% Credited It — AlexTamkin · 2026-10-28
• Not Low Effort: Analyst Read Materials 40+ Minutes, Then Told the Agent 'Use Your Recommendation' — AlexTamkin · 2026-10-28
• Agent Users Finished Faster With Far Fewer Serious Errors — That's Not the Worry — AlexTamkin · 2026-10-28
• Tamkin: Unnoticed Deference May Make 'Human in the Loop' Far Less Reassuring — AlexTamkin · 2026-10-28
• Caveats: The Lean Was Deliberately Strengthened; Everyday Effect Likely Smaller — AlexTamkin · 2026-10-28
• Alex Tamkin: Institutions must redesign themselves to use AI while preserving human judgment — AlexTamkin · 2026-10-28
• Anthropic researcher: agents shape human decisions; design models and products to protect agency — AlexTamkin · 2026-10-28

2 near-duplicate retellings: AlexTamkin · AlexTamkin