FINER: MLLMs Hallucinate under Fine-grained Negative Queries
Rui Xiao, Sanghwan Kim, Yongqin Xian, Zeynep Akata, Stephan Alaniz
CVPR 2026
cs.CV, cs.AI
2026-03-18
FINER probes MLLMs with fine-grained negative queries. InternVL3.5-14B paired accuracy on CompreCap Multi-rel rises from 47.0% to 71.2% (+24.2).
Multimodal models can still reject a coarse question such as whether a wolf is in the image. Once the query stacks several objects, attributes, and relations, and swaps one of them for a plausible detail that is not there, the model often agrees. POPE and DASH test single-object presence. AMBER's discriminative split still uses one object, one attribute, or one relation at a time. Fine-grained, multi-element negative queries barely show up in existing suites.
The authors call this setting Fine-grained Negative Queries. A motivation study builds seven granularity levels, each injecting one contradiction, with the correct answer always No. InternVL3.5-14B falls from about 80% at level 1 to about 20% at levels 5 to 7 on FINER-CompreCap, and from about 58% to about 15% on FINER-DOCCI. Finer queries produce more false positives.
Both benchmarks start from scene graphs. FINER-CompreCap uses human scene graphs from CompreCap on COCO images. FINER-DOCCI extracts objects, attributes, and relations from DOCCI's long human captions with Gemini-2.0-Flash, then filters with Qwen2.5-VL-72B. Each positive entity gets four negatives that are semantically plausible but absent from the image. A strong MLLM acts as a discriminator and regenerates ambiguous negatives so the substitute is less likely to actually be in the picture.
Four settings:
Items are five-way multiple choice. The correct option names what is actually in the image; the other four are remaining negatives of the same type. Each negative question is paired with its positive twin. Accpaired counts a pair only when both answers are right, which blocks a strategy of always saying No. CompreCap has 6300, 3338, 4280, and 3166 questions across the four settings; DOCCI has 10000, 28630, 11542, and 20944.
FINER-Tuning builds preference data in the same spirit and trains with DPO. Images come from Pixmo long captions, excluding COCO and the DOCCI training split. Annotation uses Phi-4-14B so the teacher is not the same LLM that built the benchmark. Every sample has a positive query and a negative query. The chosen answer starts with the correct Yes or No and then states the true visual details; the rejected answer does the opposite. Training uses up to 160k preference tuples, LoRA for one epoch, β=0.1, on LLaVA-1.6-7B, Qwen2.5-VL-7B, InternVL-3.5-8B, and InternVL-3.5-14B.
Prior hallucination-tuning recipes barely move on this suite. LLaVA-RLHF and RLHF-V score 1.1% and 1.6% paired accuracy on CompreCap Multi-rel. RLAIF-V-12B is the strongest of that group at 62.2% Multi-obj, still below untuned Qwen2.5-VL-7B at 69.2%.
| Model | CompreCap Multi-rel | After FINER-Tuning | Change |
| InternVL-3.5-14B | 47.0% | 71.2% | +24.2 |
| InternVL-3.5-8B | 49.8% | 64.1% | +14.3 |
| LLaVA-1.6-7B | 7.6% | 24.2% | +16.6 |
| InternVL-3.5-38B (untuned) | 66.8% | n/a | n/a |
The tuned 14B model beats its own 38B sibling by 4.4 points on Multi-rel. Wh remains hard: InternVL-3.5-38B scores 36.6% on DOCCI Wh, Gemini-2.5-Flash 49.6%. Harder counts hurt more, and the tuning gain grows with them: on CompreCap the 14B model gains 8.3, 19.1, and 28.1 points at 6 objects, 3 attributes, and 3 relations.
The same checkpoints also improve eight existing hallucination suites. InternVL-3.5-8B rises from 68.3% to 74.5% on DASH (+6.2); the 14B model from 55.8% to 61.3% (+5.5). LLaVA-1.6 gains 19.3 on HaloQuest (44.2 to 63.5) and 6.9 points on AMBER (78.1% to 85.0%). On six general multimodal benchmarks the 14B average moves from 68.0 to 69.4. RLAIF-V, by contrast, drops OmniLMM from 46.3 to 45.7 on that average. The usual alignment tax does not show up here.
Ablations match the recipe to the data. SFT on the same pairs drops InternVL-3.5-8B Multi-obj from 74.2% to 37.5%. DPO on negative queries alone already beats the base model; training on both positive and negative queries is the full setting. Training on one subset peaks on that subset; mixing all four is the most balanced.
Hallucination tests for MLLMs have lived on object presence. Users ask sentences with a chain of details, and the error is often a single swap. FINER turns that gap into a paired metric, then shows that DPO on fine-grained negative queries can lift the new suite, older hallucination benches, and general skills together. The recipe is ordinary: Phi-4-14B can write the data, without iterative closed-source multimodal annotation.
Wh questions and high-granularity items stay far below a level where saying No should be easy. Treat the suite as a diagnostic, not as a solved failure mode.
The large benchmark is not fully human-validated. Template MCQs can sound unlike natural questions. Multi-rel stops at three relations. Negatives and DOCCI scene graphs both pass through LLMs, so residual noise mixes "the model failed" with "the item is fuzzy". Gemini-2.5-Flash was only scored on 3k questions per DOCCI setting. Accpaired is strict: one miss zeros the pair, so it is not comparable to ordinary accuracy. The SFT collapse is a warning that this data fits preference learning; imitating the chosen answer can teach the wrong habit.