Annals paper: human-in-the-loop in clinical AI is now an illusion of liability, not oversight
EricTopol · x · 2026-10-27
Eric Topol, Pranav Rajpurkar, and colleagues published an Ideas and Opinions paper in Annals of Internal Medicine, "The Illusion of Oversight in Clinical AI: Human Out of the Loop?", arguing that human-in-the-loop review in clinical AI persists nominally while dissolving operationally.
- Context: In April 2026, the Utah Medical Board suspended the first US pilot letting an AI legally renew chronic prescriptions; its safety case rested on per-referral physician review, yet the design shifted to retrospective sampling.
- Functional test of autonomy: if clinician inaction still makes AI output real (prescriptions filled, diagnoses documented), the system is effectively autonomous. One insurer's utilization-management AI averaged 1.2 seconds of physician time per case; a major ECG service performs 160,000+ interpretations yearly, with 60,000 algorithmic denials signed off in a single month — per-decision review is mathematically impossible.
- Automation bias is measurable: randomized trials show trained physicians receiving erroneous AI recommendations score lower (73.3% vs 84.9%); drug-interaction alerts are overridden 90% of the time; AI-drafted patient replies increased read time by 21.8% without cutting reply time.
- Why ECG precedent doesn't transfer: ECG algorithms produce structured outputs with well-characterized failures and retrospective recoverability; LLM systems emit open-ended language, encode demographic biases that review can't catch, and can be adversarially manipulated.
- Conclusion: today's human-in-the-loop is a mechanism for distributing liability, not safety — manufacturers are incentivized to keep it nominally. The authors propose three conditions for meaningful oversight (bounded outputs, retrospectively recoverable failures, review volumes proportionate to claims) and validation thresholds anchored to operational behavior rather than nominal classification.
Related event: Topol Paper Warns Human Oversight of Clinical AI Is an Illusion(2 posts)→
More from Safety
- Journalists comb through Anthropic's risk filings before sunrise — nmasc_ · 2026-10-27
- Delip Rao asks: do we know if data sent to closed models gets intercepted? — deliprao · 2026-10-27
- Open-source cyber models now require KYC, sparking debate over gated 'open' weights — GabGarrett · 2026-10-27
- ACL ARR's controversial 'review withholding' rule draws criticism from Delip Rao — deliprao · 2026-10-27
- Open-weight model poisoning and sleeper-agent risk gain attention — ianand · 2026-10-27
- Maze Launches Maze Code: AI Agents That Investigate Dependency and Code Vulnerabilities — AndrewM_Webb · 2026-10-27