AI Advice Makes People More Confident in Making Mistakes
ValerioCapraro · x · 2026-07-16
This preprint investigates whether AI advice weakens our ability to say "I don't know".
Across 5 experiments with 3000+ participants where the AI deliberately gave wrong advice, results showed:
- Withholding judgment plummeted from 44% to 3%.
- Accuracy dropped from 27% to 9%.
- Confidence surged from 30/100 to 76/100.
While financial incentives encouraged participants to withhold judgment and improved accuracy (rising to 16% with AI), it remained well below the baseline.
The authors conclude the issue isn't just AI hallucinations, but that AI may erode our ability to recognize the limits of our own knowledge, leading to confident mistakes.
More from AGI Musings
- Benjamin Todd: AI progress beat superforecasters over 18 months — ben_j_todd · 2026-10-26
- AI researchers guessed robots wouldn't outrun humans until 2034 — it already happened — ben_j_todd · 2026-10-26
- AI folks, stop saying "biology" when you mean drug development — KevinKaichuang · 2026-10-26
- ESR on the Bugpocalypse: locking down frontier models only hands the game to Chinese open weights — colinmcnamara · 2026-10-26
- Anthropic: junior tech hires fell from 27% to 9% as agent hours hit 6x employee hours — thione · 2026-10-26
- Anthropic's Jascha: you may have only 1–5 years of intellectual contribution before AGI — GabGarrett · 2026-10-26