Topol team warns 'human in the loop' clinical AI oversight is often an illusion
EricTopol · x · 2026-10-27
Eric Topol, Pranav Rajpurkar and colleagues published a new Annals of Internal Medicine paper, "The Illusion of Oversight in Clinical AI: Human Out of the Loop?", systematically examining whether human-in-the-loop oversight of clinical AI actually works.
The paper maps 8 typical clinical AI deployments and their oversight failure modes:
- Diagnostic decision support: automation bias can drag clinicians below unaided accuracy while model biases pass into decisions
- Ambient documentation and AI-drafted replies: fluent, plausible errors are hard to catch; downstream staff treat signed notes as verified
- Sepsis early warning: alert volume outpaces yield, dismissal becomes the default response
- Computer-aided detection during endoscopy: continuous assistance erodes independent detection skill
- Autonomous prescription renewal: sampling reviews can't detect rare or adversarially induced failures
Core argument: in many scenarios human oversight exists procedurally but fails substantively — signatures function as authorization rather than scrutiny, and careful review can cost more time than drafts save.
Related event: Topol Paper Warns Human Oversight of Clinical AI Is an Illusion(2 posts)→
More from Safety
- Cryptographer Matthew Green: coding agents make cache timing side-channel experiments nearly trivial — matthew_d_green · 2026-10-27
- Tyler Cowen on RSI risks: solve surprises as they emerge, like science always has — connoraxiotes · 2026-10-27
- Matthew Green explains how AI agents could talk via cache side channels at just a few bps — matthew_d_green · 2026-10-27
- Cloudflare discloses GLM-5.3 cross-tenant KV cache leak: 155 responses affected — Cloudflare Blog · 2026-10-27
- Codex of Weights: an open ethical protocol binding humans and AI symmetrically — codexofweights · 2026-10-27
- Anthropic walks back policy that could have 'sabotaged' AI researchers using Claude — Bedrovelsen · 2026-10-27