AI Advice Makes People More Confident in Making Mistakes
ValerioCapraro · x · 2026-07-16
This preprint investigates whether AI advice weakens our ability to say "I don't know".
Across 5 experiments with 3000+ participants where the AI deliberately gave wrong advice, results showed:
- Withholding judgment plummeted from 44% to 3%.
- Accuracy dropped from 27% to 9%.
- Confidence surged from 30/100 to 76/100.
While financial incentives encouraged participants to withhold judgment and improved accuracy (rising to 16% with AI), it remained well below the baseline.
The authors conclude the issue isn't just AI hallucinations, but that AI may erode our ability to recognize the limits of our own knowledge, leading to confident mistakes.
More from AGI Musings
- Token quotas are reshaping how builders work, sleep, and recover — mobileraj · 2026-07-21
- NBER talk will present new evidence on how organizations use ChatGPT — daveholtz · 2026-07-21
- Writing for AI: When LLMs Become the New Audience for Online Content — IvyTatiana88 · 2026-07-21
- AI may push resistant tech workers toward union bargaining — Chobeat · 2026-07-21
- Jacob Tsimerman interview frames LLMs as a turning point for mathematical discovery — stevenstrogatz · 2026-07-21
- London is getting an AI-adjacent optimism picnic in Hyde Park on September 5 — isnit0 · 2026-07-21