LLM diagnostic accuracy drops 60% when patients communicate vs doctors
wandedob · x · 2026-08-23
A randomized control study reveals that when patients act as the communicator rather than physicians, an LLM's diagnostic accuracy drops by 60%, with a 12% drop in appropriate management decisions.
The findings confirm several key suspicions:
- LLMs are suggestible, naive, and sycophantic.
- Output quality is driven by medical literacy; lower literacy leads to less accurate outputs.
- Prompting still confounds the medical accuracy of frontier models.
It is considered unsafe for people to trust LLM medical advice when they are incapable of detecting the model's biases and inaccuracies.
More from Safety
- Opinion: Data Center Bans Reflect Public Powerlessness — edelwax · 2026-08-23
- AliExpress caught using silent audio waves to fingerprint devices — SumitGup · 2026-08-23
- 2050 Scenario: Fearmongering and Overregulation Drive Industry Offshore — beffjezos · 2026-08-23
- Dual control for high-risk agent tools: a production governance cheatsheet — blaizedsouza · 2026-08-23
- Reflecting on Alondra Nelson's "Thick Alignment" Keynote from 3 Years Ago — arvindsatya1 · 2026-08-23
- How to Actually Verify AI Privacy Claims? Industry Lacks Verification Mechanisms — Sea_Supermarket_8755 · 2026-08-23