A 52-minute documentary tests whether chatbots know when they don't know
CielCouvert · reddit · 2026-11-14
The author made a 52-minute documentary on whether AI chatbots have metacognition — knowing about their own knowledge.
- Testing an open model on 600 trivia questions: it claimed "100% sure" on two out of three answers, and a quarter of those were wrong. Humans are right only about three in four times when saying "100% sure", a strikingly similar pattern.
- But the doubt was visible inside the model: reading its internal signals before it answered predicted its upcoming errors far better than its own words.
- In short, the AI knows more than it says.
- Practical tip: don't trust a confident tone. For anything important, re-ask the same question in a fresh chat — when it truly knows, answers agree; when it's confabulating, they drift.
More from Models
- SubQ Snap 1.2 gains 5 points to tie Drex Max atop Artificial Analysis Decision Index — ArtificialAnlys · 2026-11-14
- Artificial Analysis Coding Agent Index v2.1: Claude Models Tie for Lead at 58 — ArtificialAnlys · 2026-11-14
- 4B specialized model hits 93.8% on bank support, beating proprietary models' 74% — MParakhin · 2026-11-14
- AI Builder Pushes Back on AGI-2027: Jaggedness Persists, Models Still Ignore Simple Instructions — ChrisGPT · 2026-11-14
- Claude models decide by emotivism but rationalize post-hoc, researcher observes — Sauers_ · 2026-11-14
- Heavy user's first impressions: Fable 5.2 is the best model, but Opus 5.5 makes more sense for heavy token users — jarrodwatts · 2026-11-14