HalluWorld Benchmark, Accepted at NeurIPS, Shows Models Nail Perception but Fail Simulation
xennygrimmato_ · x · 2026-09-29
HalluWorld is a controlled benchmark and methodology placing LLM agents inside fully specified reference worlds—grid worlds, chess, and terminal tasks—to verifiably measure and induce hallucinations. Accepted at NeurIPS 2026 Evals & Datasets.
Key findings:
- Perception is near-solved: 7 of 12 models hit 0.0% perceptual error in grid worlds; 5 reach 0.0% on chess without FEN provided.
- Simulation is not: every model hallucinates on at least 16.7% of grid memory probes, and the best models still fail 18–32% of causal chess probes when a wrong FEN is in context.
- Abstention lags: no model gets below 24% hallucination on grid uncertainty probes; realistic terminal tasks show 9–27% uncertainty hallucinations while other categories approach zero.
The takeaway: hallucination is not one capability—perception, causal simulation, memory, and uncertainty need separate measurement.
More from coding & agent
- Continual-ARC: lifelong ARC-AGI stream finds agent harness persistence rarely beats simple caching — unifyai · 2026-11-16
- WebArena-Pro: a 300-task multimodal benchmark across 20 self-hosted apps for web agents — McGill-NLP · 2026-11-16
- Filmworkbench v0.1.2: one-prompt semi-automated short films with Claude and ComfyUI — Eastern-Ad9515 · 2026-11-16
- Stop prompting LLMs for JSON: PowerShell module Jev routes with decision models — dfinke · 2026-11-16
- llama.cpp offloads CI caches and runners to Hugging Face to survive 35 PRs/day — ggerganov · 2026-11-16
- I killed my 30-minute weekly sync with a decision log my agents read before every run — Humaux · 2026-11-16