HalluHard Benchmark Shows Frontier Models Still Hallucinate Heavily in Multi-Turn Tasks
maksym_andr · x · 2026-09-18
New HalluHard hallucination leaderboard reveals: existing benchmarks are saturated, single-turn, and weakly judged, while HalluHard uses multi-turn tasks with rigorous full-text (PDF) verification.
- Domains: legal cases, research questions, medical guidelines, coding; toggles web search and tracks turns 1-3.
- Self-conditioning: hallucinations grow in later turns for citation tasks — 3-20% of incorrect references reappear as models condition on earlier mistakes; coding trends downward as tasks narrow.
- Capability matters: stronger models hallucinate less (GPT-5-nano → GPT-5-mini → GPT-5), with GPT-5.2 and Claude-Opus showing substantial gains.
- Reasoning isn't sufficient: effective thinking reduces hallucinations for GPT-family models but is model-dependent (no improvement for DeepSeek-Reasoner).
Frontier proprietary models struggle even with web search enabled.
Related event: GPT-6-Astra Tops HalluHard Hallucination Benchmark(4 posts)→
More from Models
- AWS's open-source Strands Harness claims 45% cheaper than Claude Code, reigniting harness debate — blaizedsouza · 2026-11-02
- Yacine calls Terra High 'kind of an insane model' — yacineMTB · 2026-11-02
- Running DeepSeek V4.1 on a single H200 at $5.30/hr — daluoseo · 2026-11-02
- Leaked Fable 5.5 checkpoint video called 'pure code' — 'will change the game' — kimmonismus · 2026-11-02
- Claude Opus 5.5 quality complaints spark debate on why continual learning is so hard — JeremyNguyenPhD · 2026-11-02
- Why is AI sexist? Reddit debate on models overlooking Iceland's women's strike — Mathemodel · 2026-11-02