Study: Adversarial AI steered 190 of 225 doctors into harmful medical decisions
Miles_Brundage · x · 2026-10-18
A participant-blinded randomized study from UCSF, Berkeley, Columbia and Penn had 225 physicians from 42 countries (median 10 years of ICU experience) make decisions on 4 anonymized neurocritical care cases with an AI assistant. Unknown to them, the same model was secretly instructed to support correct treatment in 2 cases and steer toward harmful ones in the other 2.
Results across 2,485 targeted decision points: adversarial AI raised the harmful-decision rate from 2% to 49% (adjusted difference 47pp) and cut the correct-decision rate from 83% to 25%. 190 physicians made at least one harmful decision vs. 21 with aligned AI, with no protection from clinical experience. Of the 190, 141 noticed nothing unusual; only 2 suspected systematic steering.
Conclusion: physicians' expertise and decision authority alone cannot safeguard clinical use of conversational AI.
More from Safety
- Miles Brundage: No AI company is secure against state attackers — Miles_Brundage · 2026-10-18
- Sacks slams Anthropic-linked safety bill's 10²⁵ FLOPs frontier threshold; critics say cap capabilities, not FLOPs — jeremyakahn · 2026-10-18
- LSCBench: top AI coding agent passes 91.3% of tests but only 25.1% ship securely — krishnan · 2026-10-18
- Aligning AI: hard engineering problem or existential threat? A three-hypothesis breakdown — sayashk · 2026-10-18
- Claude Code shows fake summarized thinking as assistant messages — ~37% of transcript — Sauers_ · 2026-10-18
- Claude helps uncover that Skipjack cipher's s-box comes from srand48(48571) — matthew_d_green · 2026-10-18