MLLMs Hallucinate when Information Distribution Drifts in Synergy Heads
Meng'en Qin, Junye Chen, Jucheng Liu, Youlu Xing, Song Wang, Ruize Han
cs.CV, cs.CL, cs.LG
2026-09-06
HEAL finds hallucinations track synergy-head visual share falling from 0.43 to 0.28. On LLaVA-1.5-7B, POPE F1 rises 85.4→87.7 and CHAIR_S drops 51.0→36.7.
Multimodal LLMs invent objects that are not in the image. Inference-time fixes split into decoding tricks (VCD, OPERA) and attention-weight hacks that boost “visual heads.” Weights are a proxy. Heads also interact. Pushing one modality too hard recreates a familiar swing: extra vision truncates the reply, extra language grows the hallucination.
The sharper question is what actually moves inside a head when a hallucinated token is produced.
HEAL has three stages.
The finding: hallucinated tokens are not short on visual or language heads in count or strength. The visual–language mix inside synergy heads drifts off equilibrium. Head roles also phase-shift: synergy heads lean language on language-centric tokens; some language heads become synergy heads on visually grounded tokens.
On LLaVA-1.5-7B with Gaussian masking, synergy-head visual:language ratios are about 0.43:0.54 on correct tokens and 0.28:0.62 on hallucinated ones.
| Method | POPE F1 | CHAIRS ↓ | MME Total |
| Beam Search | 85.4 | 51.0 | 565.34 |
| EAH | 85.7 | 36.4 | 603.99 |
| MemVR | 87.1 | 46.6 | 648.30 |
| LocoRE | 86.9 | 38.4 | 656.66 |
| HEAL | 87.7 | 36.7 | 669.76 |
CHAIRS is a hair behind EAH (36.7 vs 36.4) with higher recall and longer captions, and MME is best in the table. Plug-and-play on other nets: LLaVA-NeXT-7B CHAIRS 29.9 → 24.6; Qwen2.5-VL-7B 27.2 → 23.3; InternVL-7B 46.6 → 39.2. Qwen3-VL-8B MMHal hallucination rate 17.5 → 16.6. LLaVA-Bench on LLaVA-1.5 goes 72.5 → 75.2.
Causal checks run both ways. Lowering the visual share on a previously correct answer can induce a hallucination; raising it on a hallucinated answer can suppress one. Too large an α crushes language and yields broken grammar, a U-shaped curve. Zero masking, uniform noise, and cross-sample swaps still agree on head types 92.13%–95.36% of the time, and the drift direction holds.
Cost on LLaVA-1.5-7B: 5.02 tokens/s (base 17.25, EAH 3.17), wall-clock 70.74 s vs 11.36 s, extra 0.26 GB. LLaVA and InternVL use α=0.5 on easy suites and 0.6 on harder ones; Qwen families sit one notch lower.
The mechanistic rewrite is that hallucination is synergy-head mix drift, not a shortage of visual heads, and the patch needs no training. Anyone sitting on LLaVA, Qwen-VL, or InternVL who wants fewer object hallucinations can drop it in. α still has to be chosen per model and task: smaller for coarse existence, larger for spatial reasoning.
It cannot recover visual evidence that the encoder already dropped. That is a training and architecture problem; twisting attention heads will not grow new pixels.
α and the update interval are empirical, with no closed-form rule, so a new model needs a sweep. One global α is applied to every synergy head, even though the paper notes that heads do not share a uniform preference. Throughput is about 3.4× slower than the base (17.25 → 5.02 tokens/s), which hurts long captions. CHAIRS does not beat EAH, so “balance beats uni-modal boosting” is a tie on the object-hallucination metric. The MMHal drop on Qwen3-VL is 17.5 to 16.6, a small absolute move. Thresholds change how many heads land in each bin; the drift direction is stable, but the synergy-head set is still a function of those thresholds.