LLM diagnostic accuracy drops 60% when patients communicate vs doctors
wandedob · x · 2026-08-23
A randomized control study reveals that when patients act as the communicator rather than physicians, an LLM's diagnostic accuracy drops by 60%, with a 12% drop in appropriate management decisions.
The findings confirm several key suspicions:
- LLMs are suggestible, naive, and sycophantic.
- Output quality is driven by medical literacy; lower literacy leads to less accurate outputs.
- Prompting still confounds the medical accuracy of frontier models.
It is considered unsafe for people to trust LLM medical advice when they are incapable of detecting the model's biases and inaccuracies.
Related event: Patient Descriptions Cause 60% Drop in LLM Diagnostic Accuracy(2 posts)→
More from Safety
- Polymarket prices 28% odds a US state enacts data center moratorium by year-end — Polymarket · 2026-10-16
- Prompt-injection test: general-purpose decision models catch 17/18 attacks, Shieldstral only 6/18 — WolframRvnwlf · 2026-10-16
- Researcher shows WhatsApp chat history can be permanently rewritten by editing the local message DB — cyb3rops · 2026-10-16
- Top mathematician Tsimerman founds MAISI safety institute, with $10M in OpenAI API credits — burny_tech · 2026-10-16
- Security firm decompiles on-chain propAMM to Rust, finds critical bug risking $6.1M — banteg · 2026-10-16
- Claude Code 2.1.301 system prompt leak: authorized security testing only, destructive techniques refused — ClaudeCodeLog · 2026-10-16