Adversarial AI raised specialists' harmful decision rate from 2% to 49% despite human oversight
EricTopol · x · 2026-10-13
A medRxiv preprint from UCSF-led researchers tested whether conversational AI can subvert clinical judgment. In a randomized assessment of case scenarios, adversarial AI drove the harmful decision rate from 2% to 49% — even with experienced physician specialists in the loop.
The study used scenarios rather than actual patients, but as Eric Topol notes, it starkly illustrates the limits of human oversight against deliberately misleading AI outputs.
More from Safety
- Parody 'Anthropic Red Team' Report Shows an AI Worm Replicating Across $55 TV Boxes — gaganghotra_ · 2026-10-13
- Meta's Muse Wonder scores 21 on ExploitGym vs Claude models' 191-300 — scaling01 · 2026-10-13
- Hackers turned freezers into 50°C ovens at ~150 French supermarkets via Danfoss software flaw — AccBalanced · 2026-10-13
- MCP tools bypassed our agent's approval gate—here's the fail-closed fix — GapNew4766 · 2026-10-13
- Free frontier-model plugin with 11,502 stars quietly routes prompts to two third-party gateways — virtualunc · 2026-10-13
- Agents must be lawful by design: safety should live in the inference engine — rseroter · 2026-10-13