Multi-domain training mix boosts supervised hallucination detectors 7.25% on out-of-distribution QA
MaartenSap · x · 2026-10-25
Presented at EMNLP, this paper systematically compares supervised classification vs. unsupervised metric-based hallucination detection across 11 datasets and 10 models, finding supervised methods win in-domain but lag out-of-domain.
To close the generalization gap, the authors propose a simple fix: train supervised detectors on a curated multi-domain data mix, optionally complemented with task-specific data. On 697K QA samples from 12 open-source QA datasets, supervised methods with this general training surpass unsupervised methods by +7.25% on OOD short-form QA without any task-specific data. Scaling analysis shows models with general data need up to 40.3% less task-specific data to approach optimal performance.
More from Research
- Safety researcher: verification should go all the way down to Maxwell's equations, not stop at assembly — JacquesThibs · 2026-10-25
- Local 35B authored 800 math problems under cloud-agent supervision — authoring erred 5x more than cold-solving — Individual_Page9676 · 2026-10-25
- SA-LapNet fixes spin exactly in neural-network quantum Monte Carlo, breaking the purity-energy trade-off — bravo_abad · 2026-10-25
- XYBench: Strongest LLMs Answer Literally and Detect Under 63% of User Misconceptions — MaartenSap · 2026-10-25
- EVALUESTEER: best reward models fail to align with user values over 25% of the time — MaartenSap · 2026-10-25
- Ouvia study: only about half of real-world speech translation interactions rated usable — MaartenSap · 2026-10-25