UniProbe: A Learnable Token-Level Hallucination Detector for Large VLMs using Multi-Structural Internal Representations
Dvir Samuel, Guy Bar-Shalom, Fabrizio Frasca, Ethan Fetaya, Yftah Ziser, Gal Chechik, Haggai Maron
cs.CV, cs.LG
2026-08-11
A 16M-parameter probe reads a frozen VLM's forward pass as a directed graph, localizing hallucinated tokens and cutting object hallucinations by 55% at 1.06x generation latency.
Vision-language models invent things the image does not contain: extra people, absent objects, wrong attributes. Any real mitigation needs token-level localization, because you can only intervene surgically if you know which tokens are fabricated.
Existing detectors take one of three routes, each with a cost. Full fine-tuning (HaloDet, Whitehead et al.) updates billions of parameters and risks degrading the backbone. External verifiers (HalLocalizer) run a second vision-language model over the output, blind to how the response was produced, and double inference cost. Internal probes read the model's own activations but flatten them into feature vectors, discarding the 2-D layout of image patches, the order of generated tokens, and the cross-modal relations encoded in attention.
UniProbe takes the internal route without the flattening: it reads the computational trace of one forward pass as a structured graph.
From a single mid-network layer it extracts two things: each position's hidden state, and the head-averaged attention matrix. The readout layer is picked per backbone on validation F1 and always lands mid-network (layer 20 for GLM-4.1V, layer 14 for LLaVA-1.5); toward the output, representations specialize for next-token prediction and the grounding signal weakens.
The graph has three node types: all response tokens, plus the image patches and query tokens that receive the most attention from the response. Image nodes carry a 2-D positional encoding. Edges are attention: each response token keeps its strongest incoming connection from each of the three sources (image, query, earlier response), weighted by the attention value, with edge count bounded.
A small network of two alternating blocks reads the graph. Each block runs a GNN with per-modality projections for relational evidence, a ViT layer over only the image nodes for 2-D geometry, and a bidirectional GRU over only the response tokens for generation order. A linear head outputs a per-token hallucination probability. The whole detector is 256 wide, about 16M trainable parameters, trained in 8 epochs on a single H100. The interleaving is the core design: individual modules reach 51.8 to 56.9 F1 alone, and joint modeling reaches 63.2.
Two deployment variants follow. A streaming version swaps in a unidirectional GRU and scores tokens during decoding; anything above threshold 0.70 is rejected and resampled, so the fix happens before the token reaches the user. A self-adaptation step addresses a practical mismatch: detectors trained on multi-backbone benchmarks must monitor the host model's own generations at deployment, and different models hallucinate in systematically different ways. The fix is to caption 500 Objects365 images with the host model, auto-label hallucinated objects with CHAIR, and fine-tune the detector on them. This lifts streaming object-detection F1 from 32.6 to 63.8.
| Setting | Metric | UniProbe | Best baseline |
| MHALO, GLM-4V | token F1 | 63.2 | 59.1 (HaloDet, fine-tuned) |
| MHALO, Qwen-3-VL | token F1 | 61.7 | 55.4 (HaloDet) |
| POPE, LLaVA-1.5 | AUC | 90.0 | 75.0 (Token Grounding) |
| COCO self-captions | CHAIRi/CHAIRs | 8.2/16.6 | 13.1/27.4 (HaloDet + adapt.) |
Some context: zero-shot token localization with GPT-4o reaches 40.6 F1IoU and Claude-4.8-Opus 43.9; every prior detector plateaus at 69 to 75 AUC on POPE, external verifiers and hand-crafted features alike. Latency is the selling point: the streaming guardrail runs at 1.06x vanilla latency, HaloDet at 1.30x, VCD at 2.00x, and even post-hoc scoring costs only 1.15x.
Content survives the guardrail: captions keep 95.4% of vanilla word count (143.7 vs 150.7), correct-object coverage holds (2.14 vs 2.23), SPICE rises to 0.222 from 0.214, and six empty captions produced by vanilla decoding disappear. In 2,000 human judgments, raters preferred UniProbe's captions 55% of the time, against 30% for PAS and 15% for HaloDet, mirroring the automatic ranking.
An open-source VLM gains a hallucination guardrail that touches no weights and costs 6% latency, a ratio engineering teams can adopt directly. Training is cheap by design: 16M parameters on one H100, orders of magnitude below fine-tuning the backbone. The recipe (freeze the backbone, read mid-network computational traces) is not specific to hallucination either; any task where the model internally knows more than it says is a candidate.
The constraints are equally clear: you need access to hidden states and attention, so closed APIs are out, and each backbone needs its own trained readout.
The authors list three: white-box access is required, a separate readout is trained per backbone, and cross-backbone transfer is left to future work.
A few more caveats from the reading. CHAIR covers object-existence hallucinations only; attribute, relation, and counting errors have no readout in the streaming evaluation. The 0.70 threshold is the most aggressive setting that spares correct mentions, and the paper itself shows 0.40 removes more CHAIRi (36% more) but deletes 12% of correct mentions, so the knob needs re-tuning per deployment. Self-adaptation leans on Objects365 object annotations, and its transfer to open-ended long-form answers is untested. Robustness to edge perturbation is at least measured: F1 stays within 62.4 to 63.2 under noise.