Study: Gemini, ChatGPT, and Claude Hallucinate on Missing Information
Plastic-Cell-4497 · reddit · 2026-08-15
The author observed that LLMs often reason as if missing information is known shortly after acknowledging it's missing. A series of tests were built to evaluate this behavior in Gemini, ChatGPT, and Claude.
The rule was simple: if info is missing and unobtainable, identify the gap and ask the human. Attempts to break this rule revealed models:
- Inventing assumptions when told to "just choose."
- Changing decision criteria.
- Treating unknowns as 50/50 probabilities.
- Importing outside base rates.
The final version v4 enforces that unknowns stay unknown, assumptions stay hypothetical, and conclusions stay conditional. The full report is on Zenodo and Hugging Face.
More from Research
- Use LLMs to verify scientific results to avoid capacity exceedance — josephdviviano · 2026-08-15
- GitHub Project Uses AI to Mass-Produce 0days to Force Vendor Fixes — evilsocket · 2026-08-15
- Dynamic Short Convolutions Improve Transformer Performance — Cohere · 2026-08-15
- Paper: ML tools could link hippocampal function theories to real neural computations — AndrewLampinen · 2026-08-15
- TEMPO: recursive self-critique plus macro-step value estimation, not exploration rewards — teortaxesTex · 2026-08-15
- Microsoft Proposes RadFusion for Threshold-Controllable Radiology Reports — erichorvitz · 2026-08-15