Beyond Language Priors: Diagnosing and Fixing Visual-Origin Hallucinations in Multimodal LLM
Peiyang Xu, Xiaopei Zhu, Jun Zhu, Xiaolin Hu
cs.CV
2026-09-01
Tsinghua traces short-answer object hallucination to visual misalignment (0.158 vs -0.122) and lifts LLaVA POPE-Adversarial from 0.779 to 0.841 with ACFT, no extra inference cost.
Object hallucination in multimodal LLMs is usually blamed on language priors: co-occurrence statistics, hallucination attention heads, summary-token sinks. That account fits long captions, where text context amplifies linguistic bias. Short answers are a different regime. Asked "is OBJ in the image?", the model leans on vision. Treating those errors as a language-head problem sends the fix to the wrong place.
On LLaVA v1.5 the paper measures two visual signatures. Mean image-text cosine similarity is 0.158 on correct samples and -0.122 on hallucinated ones. Smooth Grad-CAM attention is inverted: when the object is present, hallucinated cases are more diffuse (entropy +5.1%); when it is absent, they concentrate on the wrong region (entropy -6.2%). Intervening on the visual encoder moves the number: Gaussian noise, downsampling, or a weaker encoder drops mean POPE accuracy from 0.842 to 0.739–0.822; a stronger SigLIP-SO400M encoder raises it to 0.864.
Ordinary contrastive fine-tuning (OCFT) pairs a matched image with a random unrelated one. Feature differences are uncontrolled, so the model never learns which visual cue triggers the false answer. ACFT first builds aligned pairs with AHAF.
AHAF runs PGD inside a small perturbation ball and flips the image's hallucination attribute: a correctly answered image becomes one that induces hallucination, or the reverse. The pair differs mainly by that controlled perturbation. The same procedure is a probe. Tiny pixel changes flip the answer on clean images, which means visual features sit next to the hallucination decision boundary.
Fine-tuning combines a contrastive loss with generative cross-entropy: pull the text anchor toward the positive image embedding, push it from the negative, and keep the original generation objective. Inference adds no extra cost. The paper claims 0.9% of COCO; the OCFT comparison uses 3,000 COCO images, and the post-training comparison is budgeted at about 6k samples.
POPE, LLaVA v1.5-7B:
| Split | origin | Best inference baseline | ACFT |
| Adversarial | 0.779 | VTI 0.805 | 0.841 |
| Popular | 0.862 | OPERA 0.886 | 0.906 |
| Random | 0.885 | VCD 0.892 | 0.897 |
MiniGPT-4-13B reaches 0.771 / 0.818 / 0.843 against origin 0.700 / 0.732 / 0.792. Qwen2.5-VL-7B is already strong and still moves from 0.864 / 0.875 / 0.884 to 0.877 / 0.900 / 0.916. On MME Existence, LLaVA goes 0.950 to 0.983 and MiniGPT-4 0.850 to 0.900; Qwen is already 1.000.
OCFT scores 0.483 on Adversarial, worse than the untouched model. Random negatives scramble the decision boundary. Against SFT, LLaVA-RLHF, CHiP-DPO, and OPA-DPO at a similar data budget, ACFT is best overall on POPE and MME Existence. Description-level gains are small: CHAIRs 0.508 to 0.494. Full MME does not drop; LLaVA moves 0.728 to 0.747.
Part of short-answer hallucination is bad visual features and image-text misalignment. Contrastive decoding, external grounding rewrites, and latent steering leave the representation untouched. ACFT edits the visual embedding, adds no inference latency, and uses a small data budget.
It also makes a sharper claim: on many clean images the visual features already sit on the hallucination boundary, and a light PGD push flips them. That is more specific than "the model likes to invent objects", and it explains why random negatives barely help.
There is no dedicated limitations section. ACFT is trained on Yes/No questions; on long captions, CHAIR and ObjectHal drop only a few points, so language priors still dominate long-form output. AHAF needs white-box PGD on the target model, which is expensive, and the pairs are model-specific. Whether adversarial flips cover naturally occurring hallucination is untested against real photographic degradations. Qwen is already perfect on MME Existence, and its POPE gains are smaller than LLaVA's. The abstract says 0.9% of COCO; the experimental text also cites 3,000 images and about 6k samples, which do not line up cleanly.