AI Advice Makes People More Confident in Making Mistakes
ValerioCapraro · x · 2026-07-16
This preprint investigates whether AI advice weakens our ability to say "I don't know".
Across 5 experiments with 3000+ participants where the AI deliberately gave wrong advice, results showed:
- Withholding judgment plummeted from 44% to 3%.
- Accuracy dropped from 27% to 9%.
- Confidence surged from 30/100 to 76/100.
While financial incentives encouraged participants to withhold judgment and improved accuracy (rising to 16% with AI), it remained well below the baseline.
The authors conclude the issue isn't just AI hallucinations, but that AI may erode our ability to recognize the limits of our own knowledge, leading to confident mistakes.
More from AGI Musings
- Data engineering, not agent frameworks, is the real bottleneck for enterprise AI agents — dhruv2038 · 2026-09-11
- François Fleuret: Only Two Long-Term Futures — No Super AI, or Staying Fully Human With It — francoisfleuret · 2026-09-11
- IG reel debunking the 'winning the AI race against China' fallacy hits 500k likes — louisvarge · 2026-09-11
- Post-AI World Leaves No Room for Learning on the Job — rachittshah · 2026-09-11
- Researcher questions AI safety eval firm, citing 'blatantly sloppy' security and monitoring — Kyrannio · 2026-09-11
- AI researcher memes agent-swarm tinkering with He Jiankui's embryo-editing quote — dejavucoder · 2026-09-11