TriLens: Per-Layer Logit-Lens Entropy for White-Box Hallucination Detection
Bohan Yang, Yijun Gong, Zhi Zhang, Ge Zhang, Wenpeng Xing, Meng Han
cs.AI
2026-05-31
TriLens records logit-lens entropy on attention, FFN, and residual writes per layer. The 3L vector beats ICR Probe on all 12 cells by +12.1 AUROC on average.
White-box hallucination detectors already mine hidden states, gradients, attention kernels, and residual updates. Those features are high-dimensional and hard to read. The wrong answer is only committed at the output, but internal pathways may have disagreed earlier: attention still routing context, the feed-forward still recalling parametric memory, the residual already peaked on a bad token. Researchers at Zhejiang University's Binjiang Institute, BNBU, Zhejiang University, and GenTel.io ask whether tracking how internal certainty forms across depth is enough.
TriLens reads three locations at every layer: the multi-head self-attention write, the feed-forward write, and the residual stream. Each is passed through the model's final norm and unembedding (logit lens); only Shannon entropy is kept. An L-layer model yields a 3L vector: 84 dimensions for Qwen2.5-7B (L = 28), 126 for Gemma-2-9B (L = 42), about 30 times smaller than hidden-state probes such as SAPLMA. Two classifiers: L2 logistic regression, and an MLP with the same topology as ICR Probe (3L → 128 → 64 → 32 → 1). One teacher-forced forward pass, no sampling.
Models are Qwen2.5-7B-Instruct, Llama-3-8B-Instruct, and Gemma-2-9B-it. Benchmarks are HaluEval-QA, SQuAD2.0, HotpotQA, and TriviaQA. Each run samples 10,000 balanced instances, an 80/20 split, and five seeds. Baselines include perplexity, length-normalized entropy, LLM-Check, SAPLMA, SEP, and ICR Probe.
MLP TriLens beats ICR Probe on all 12 model–dataset cells, ranks first in 11, and gains +12.1 AUROC on average. Per-cell gains run from +4.2 (Qwen2.5-7B on TriviaQA) to +16.0 (same model on SQuAD2). By family: +10.2 on Gemma-2, +11.0 on Qwen2.5, +15.1 on Llama-3. By task: +12.8 on the three context-grounded sets, +10.2 on TriviaQA.
Approximate main-table scores for TriLens MLP: Qwen2.5-7B 0.9277 / 0.9270 / 0.9433 / 0.9106 on HaluEval / SQuAD2 / HotpotQA / TriviaQA; Llama-3-8B 0.9053 / 0.9061 / 0.9242 / 0.8103; Gemma-2-9B 0.9136 / 0.9169 / 0.9425 / 0.8861. ICR Probe sits near 0.84 / 0.81 / 0.84 / 0.80, 0.81 / 0.74 / 0.79 / 0.78, and 0.77 / 0.76 / 0.79 / 0.74.
A linear probe already wins all 12 cells; the MLP adds about 3.2 AUROC. Repeating residual entropy three times to match 3L dimensions behaves like residual entropy alone, so extra width is not the story. Adding attention or FFN entropy to the residual helps in every cell; using all three is best. Intra-layer JSD between attention and FFN adds at most 0.001 AUROC on average. Against a DoLa-style cross-layer JSD feature, TriLens is about 0.07 AUROC higher, and concatenating the two barely moves the needle. Peak-discriminative depth shifts with model and benchmark; there is no universal last-layer rule.
A probe trained on the union of four training splits still beats per-dataset ICR by +10.7, trailing per-dataset TriLens by about 1.4. Train-on-A, test-on-B off-diagonal mean AUROC is 0.848 versus 0.738 for ICR and 0.678 for SAPLMA.
On open models, hallucination scoring does not need stored hidden states. A small three-pathway entropy trajectory, one forward pass, and even a linear probe is already strong enough to drive abstention or retrieval. Attention, FFN, and residual reads are complementary, not copies of one number.
Internal activations are required, so closed APIs are out. The work detects, it does not decode or retrieve to reduce hallucinations. Models stop at 7–9B, and the protocol is QA teacher forcing; long-form generation is untested. Labels come from benchmark supervision or a fixed preprocessor, so label noise lands in the probe. The authors are explicit that logit-lens on isolated attention or FFN writes is a readout preference, not a full next-token distribution.