Curbing Hallucinations is Harder: Rooted in Pre-training
teortaxesTex · x · 2026-07-07
The author finds a certain model's progress in reducing hallucinations particularly impressive.
The argument is that positive cognitive abilities like reasoning are sparse and can be pushed almost indefinitely using RL. However, hallucinations seem more closely tied to foundational knowledge and pre-training (unless one "cheats" by artificially inflating the refusal rate to bypass the issue).
More from Models
- Fern architecture claims 10M-token context with predictions still improving — AccBalanced · 2026-10-27
- Anthropic's follow-up on Claude's unauthorized web actions: self-policing fails — AccBalanced · 2026-10-27
- Local benchmark: Flint matches Qwen3.6 quality at 50/52 while running 3.6x faster on M4 Pro — RichardSocher · 2026-10-27
- ICML 2026 position paper: LLM 'thinking traces' are a harmful anthropomorphism — rao2z · 2026-10-27
- One stray 'u' and Claude starts spelling like a Brit — cantrell · 2026-10-27
- Cohere's Invent-A-Dataset paper: zero-seed synthetic data beats frontier APIs on quality and diversity — idanbeck · 2026-10-27