HalluHard Benchmark Shows Frontier Models Still Hallucinate Heavily in Multi-Turn Tasks
maksym_andr · x · 2026-09-18
New HalluHard hallucination leaderboard reveals: existing benchmarks are saturated, single-turn, and weakly judged, while HalluHard uses multi-turn tasks with rigorous full-text (PDF) verification.
- Domains: legal cases, research questions, medical guidelines, coding; toggles web search and tracks turns 1-3.
- Self-conditioning: hallucinations grow in later turns for citation tasks — 3-20% of incorrect references reappear as models condition on earlier mistakes; coding trends downward as tasks narrow.
- Capability matters: stronger models hallucinate less (GPT-5-nano → GPT-5-mini → GPT-5), with GPT-5.2 and Claude-Opus showing substantial gains.
- Reasoning isn't sufficient: effective thinking reduces hallucinations for GPT-family models but is model-dependent (no improvement for DeepSeek-Reasoner).
Frontier proprietary models struggle even with web search enabled.
Related event: GPT-6-Astra Tops HalluHard Hallucination Benchmark(4 posts)→
More from Models
- 'You Can Feel the Gradient Slopes': Dev Contrasts Benchmaxxed RL Models vs General Ones — theteknosaur · 2026-11-02
- LightOnOCR-3 Launches: Tops FRBench, Adds Layout Structure and Chart Extraction — IgorCarron · 2026-11-02
- NVIDIA's Kumo Is a Family of Four Tabular Foundation Models, Not Just One — pandeyparul · 2026-11-02
- SemiAnalysis: safety classifiers are underrated — Meta's Muse tops cyber benchmark at 35.2% — ThomasScialom · 2026-11-02
- DeepSeek V4.1 Flash 1017 wins over a longtime closed-source-only user: fast, cheap, reliable — _bachrc · 2026-11-02
- Local quants vs Claude on 9 real private bugs: 3-bit costs thinking time, not score — FeydRowan · 2026-11-02