Can LLMs catch their own hallucinations by reading logprobs? A Reddit experiment
Any-Chipmunk5480 · reddit · 2026-08-12
A developer shared intriguing findings on LLM hallucinations, building a custom WebUI to extract logprobs via tool calling. They tested whether Gemma and Qwen could detect their own uncertainty.
Key observations:
- First recall matters most: Token probabilities during the first factual recall (before self-conditioning) are the best uncertainty indicator. Once self-conditioned on a belief, probabilities quickly shoot up to 100%.
- Rival tokens signal guessing: If candidates are distinct facts (e.g., Thomas vs Daniel), it indicates unreliable recall. Variations of the same fact (e.g., Paris vs Pari) don't.
- Uncertainty disperses probability: When models don't know an answer, probabilities scatter across multiple tokens rather than confidently locking onto a wrong one.
In practice, both Gemma and Qwen were pretty bad at actually reading their own logprobs to self-correct.
Related event: Developer Tests Logprobs to Detect LLM Hallucinations(2 posts)→
More from Research
- GPT-5.6 Assists in Proving Lower Bound for Gradient Descent Acceleration — prof_grimmer · 2026-08-12
- u-OPSD: Self-Distillation Without Labels or Teachers Beats GRPO on Math Reasoning — burny_tech · 2026-08-12
- Survey of 150+ Agent Memory Architectures: Self-Evolving Designs Boost Retention by 50% — blaizedsouza · 2026-08-12
- Stanford Paper Reveals Multi-Agent Flaws, Introduces Control Plane for 10x Efficiency — blaizedsouza · 2026-08-12
- New Book Reveals Imbalanced Data Truths: Data and Models Matter More Than Balancing — Al_Grigor · 2026-08-12
- Claude Solves FrontierMath Open Problem: Finds Hadamard Matrix of Order 668 — inductionheads · 2026-08-12