HalluWorld: Controlled LLM Hallucination Benchmark Finds Perception Solved, Simulation Still Hard
Jeande_d · x · 2026-09-29
HalluWorld is a controlled hallucination benchmark built on fully-specified reference worlds (grid worlds, chess, terminals), accepted to NeurIPS 2026 Evaluations & Datasets. A model hallucinates when it makes an observable claim false in the reference world.
Key findings:
- Perception is near-solved: 7 of 12 models hit 0.0% perceptual error in grid worlds; 5 reach 0.0% on chess without a provided FEN.
- Simulation is not: every model hallucinates on at least 16.7% of grid memory probes, and even top models fail 18–32% of causal chess probes once an incorrect FEN is in context.
- Abstention lags: no model gets below 24% hallucination on grid uncertainty probes; terminal tasks stay at 9–27%.
The paper argues hallucination is not one capability, and existing benchmarks are too fragmented to compare mitigations across settings.
More from Research
- Google Research open-sources PaperOrchestra, a multi-agent framework for writing AI papers — google-research · 2026-11-16
- Kind ICLR Reviewer Raises Score Despite Unresolved Concerns, Citing the Paper's Value — MarkSchmidtUBC · 2026-11-16
- Hand-editing 0.2% of weights plus 100 CPU-minutes turns an English TTS model into an 18MB Sanskrit one — kastnerkyle · 2026-11-16
- Why chirality matters for AI for Science: enantiomers behave differently in biology — madeofAjala · 2026-11-16
- RSS 2027 submissions open with FAQ clarifying two-stage review and negative-result bar — GeorgiaChal · 2026-11-16
- Mobile AI Factory: agents auto-build on-device SLMs with 19x less GPU time — pmttyji · 2026-11-16