RAG Silent Hallucinations: Agent Claims 'Not in Corpus' After Reading 0.6%
CupGlass540 · reddit · 2026-08-25
The author exposes a "silent hallucination" in RAG systems where an agent retrieves only a tiny fraction of the corpus (e.g., 0.6%) but answers as if it has read everything. This is critical for negative claims (e.g., "no policy exists"), where the model asserts facts about unread documents—a failure standard faithfulness scores miss because they only check answer grounding, not corpus coverage.
The author argues that claims of existence need only cite the relevant chunk, but claims of absence/universality require exhaustive coverage. They open-sourced notchecked, an MCP server that logs corpus coverage before answering and distinguishes between retrieval failures, assembly drops, and reasoning errors. The post highlights the need for "Context Recall" in production evals.
More from coding & agent
- Mini-PC for AI Agents: Seeking Hardware to Run 6-10 Concurrent Agents — koltregaskes · 2026-08-25
- Agent Firewall v1.3: Enforcing transitive authority in delegated AI agents — ShubhBhangu · 2026-08-25
- dsh-fs-deny-policy: Plugin to keep the model out of restricted folders — vladlearns · 2026-08-25
- Gradio Launches Workflow: Visual Canvas for AI Pipelines with One-Command Deploy — Gradio · 2026-08-25
- Getting LLMs to delete legacy code is as hard as for junior devs — tobowers · 2026-08-25
- UK Regulators: 'My Agent Did It' Is Not a Valid Legal Defense — iubenda_team · 2026-08-25