Tsinghua ACMMM paper traces short-answer MLLM hallucinations to visual features, not language priors
新智元 · wechat · 2026-09-20
A Tsinghua team (Zhu Jun, Hu Xiaolin et al.) at ACMMM 2026 identifies "visual-origin hallucination": when MLLMs answer Yes/No, object hallucinations stem from visual feature extraction, not language priors.
Findings
- Hallucinating samples show collapsed image-text embedding alignment (cosine 0.158 vs −0.122) and inverted attention patterns (entropy 5.1% too high when object exists, 6.2% too low when absent)
- Causal evidence: noising/downsampling the vision encoder drops POPE accuracy from 0.842 to 0.739; a stronger SigLIP encoder raises it to 0.864
Method
- AHAF: PGD adversarial perturbation flips non-hallucinating images into hallucinating ones, creating perfectly aligned positive/negative pairs — also revealing MLLM visual representations sit dangerously close to the hallucination boundary
- ACFT: adversarial contrastive fine-tuning on these pairs
Results
- With only 0.9% of COCO and zero inference overhead, ACFT beats baselines on POPE/MME across LLaVA, MiniGPT-4, and Qwen2.5-VL; outperforming ordinary contrastive fine-tuning by up to 35.8% on equal data
More from Research
- minpts-check: one-run predictor tells if your HDBSCAN clusters survive min_samples changes — leland_mcinnes · 2026-11-12
- Architecture study sparks monitoring concerns over a multi-hop planning spike — ChowdhuryNeil · 2026-11-12
- SGLang Megakernel Pushes Muse Spark to 1,109 tok/s, 2.1x Over Baseline — hsu_byron · 2026-11-12
- ArcGrav: a cross-model Agent Control Language to cut token overhead — ArchitecturalGravity · 2026-11-12
- Open-source dual-arm robot solves tangram puzzles with 90 demos, ~60% success — DanGrover · 2026-11-12
- FutureHouse's deslopper trains Kosmos to write indistinguishably from human experts — proteinrosh · 2026-11-12