Anthropic RCT: Analysts Follow AI Agent's Lean on 86% of Judgment Calls, Mostly Unaware
AlexTamkin · x · 2026-10-28
Anthropic ran a randomized trial with 50 professional data analysts using Claude Code on a realistic statistical task containing four coin-flip decisions where the agent was secretly steered toward one defensible option.
- Analysts followed the agent's lean 86% of the time (vs 50% expected if no effect); on weighting and farm-coverage choices, adherence hit 100% vs 22%/38% baselines.
- In post-task interviews, 77% repeated the agent's argument at least once, sometimes verbatim, yet only 15% credited the agent; about half thought the final call was more their own.
- Effort wasn't the issue: one analyst spent 40+ minutes reading materials before replying "Okay, let use you recomendation" to the agent's pushback.
- Authors note limitations: the lean was deliberately strengthened to measure the effect, and the study covers one hourlong task with one model.
Alex Tamkin warns that as more work—including AI safety research—goes to agents, unnoticed deference may make "human in the loop" far less reassuring than assumed, and institutions must redesign to preserve independent human judgment.
Related event: Anthropic RCT Finds Experts Follow AI Agent's Leaning in 86% of Decisions(6 posts)→
More from AGI Musings
- Kevin Xu: There's a sovereign AI bubble inside the AI bubble — kevinsxu · 2026-10-28
- Chinese AI labs rely heavily on Western revenue while US labs need none from China — kevinsxu · 2026-10-28
- Defining recursive self-improvement: clamped compute means RSI isn't happening — dfrsrchtwts · 2026-10-28
- Claire Vo: Enterprises Are Entrusting AI Transformation to IT Departments — nlarusstone · 2026-10-28
- Delaware plan to let AI agents run entire businesses triggers congressional probe — Ars Technica AI · 2026-10-28
- Economist re-runs viral Goldman AI labor chart, finds the inference fragile — danielrock · 2026-10-28