80% of MDLM hallucinations sit on entropy spikes; StepRefine only fixes those

Rethinking in Spikes: Mitigating Hallucinations in MDLMs with Step-Aware Decoding

Zhongxing Xu, Zhonghua Wang, Zhe Qian, Shiyan Su, Ming Hu, Xiaocheng Zou, Wei Feng, Mingquan Lin, Yuyin Zhou, Yifan Peng, Hamid Rezatofighi, Zongyuan Ge

cs.CL, cs.AI

2026-10-14

StepRefine refines MDLM denoising only at entropy-spike steps while keeping the forward budget fixed, and four backbones gain 2.3 to 2.8 points on average.

What problem this solves

Multimodal diffusion language models (MDLMs) do not write an answer left to right. They start from a fully masked sequence, predict every masked position in parallel, commit some of those predictions, and leave the rest masked. Bidirectional attention can see the image and the tokens already committed. Hallucinations are still common. Training-free decoding for diffusion language models has mostly chased speed, through cache reuse or step schedules, or it has treated every denoising step as the same kind of uncertainty, either by remasking weak tokens throughout or by swapping in continuous representations at every step.

Mean mask entropy does fall as denoising proceeds. The drop is not monotone. A few steps jump sharply relative to a local time window. Those steps are entropy spikes. They are rare, and on LLaDA-V and Dream-VL about 80% of hallucinated tokens land where the step-to-step entropy change exceeds the local mean plus one standard deviation. Spikes also cluster early: 71% of them on LLaDA-V fall in the first half of the trajectory, and 62% on Dream-VL. Most positions are still masked then, so the context is thin.

A perturbation on LLaDA-V separates two failure modes. Replacing low-confidence remasking (commit the positions the model is surest about) with random remasking on a random 10% of steps drops VStar, RealWorldQA, and MathVista. The same swap, applied only on entropy-spike steps, barely moves the scores. Confidence ranking works on ordinary steps. On spike steps it no longer separates candidates, and taking the mode locks a token in while the context is still weak.

Method

StepRefine leaves the weights alone and edits the decode loop. Each step runs one forward pass from mask embeddings, averages the Shannon entropy over masked positions, and subtracts the previous step's entropy. The mean and standard deviation of entropy changes inside a trailing window form a local baseline. A step whose change exceeds that mean by one standard deviation is a spike. The window is 16. Shorter windows fire on step-to-step noise. Longer windows blend stages that do not share an uncertainty level, and accuracy starts to oscillate once the window passes a moderate size.

Ordinary steps still take the highest-probability token and commit with the original remasking rule. Spike steps switch to latent refinement. The distribution stays intact and is turned into a probability-weighted average of token embeddings. That soft vector replaces the mask embedding, and the model runs forward again. Candidates that are still alive keep interacting with the image and the text already written. The loop stops when the average KL divergence between successive distributions, measured on masked positions, drops below 0.01. Only then does discrete decoding commit tokens.

The extra forwards are charged to the original budget. If a spike step uses several refinement passes, the scheduler skips the same number of later denoising steps, so the total number of forwards stays at the preset 256. Generation length is also 256, with low-confidence remasking as the default. The scheduler relocates compute onto unstable steps and keeps the forward count fixed. Forcing every spike to take 2, 4, or 8 refinement rounds never wins on both datasets at once. Stopping when the distribution stabilizes beats those fixed budgets. Inserting refinement at a fixed fraction of steps also loses to the spike trigger, because spike locations move from sample to sample.

Results

The backbones are LLaDA-V-8B, LLaDA-O-8B, Dream-VL-7B, and LaViDA-L-8B. CoTA, ReMix, and WINO, three training-free decoders, are fully compared only on LLaDA-V. On that model, StepRefine lifts average accuracy on the five general-reasoning benchmarks by 2.8 points over the base decoder. Across the 12 main metrics and all four backbones, with MMHalu and Bingo scaled by their maximum scores, the average gain sits between 2.3 and 2.8 points.

SetupMetricResult
LLaDA-V base / StepRefineVStar55.5 / 58.6 (+3.1)
CoTA / ReMix / WINOVStar57.1 / 56.5 / 56.5
LLaDA-V base / StepRefineMMVP40.7 / 45.3 (+4.6)
LLaDA-V base / StepRefineMMHalu (0-6)3.35 / 3.50
LLaDA-V base / StepRefine / WINOBingo (1-5)2.71 / 2.93 / 2.96
Dream-VL base / StepRefineVStar69.1 / 72.8 (+3.7)
Dream-VL base / StepRefineMathVision26.7 / 30.9 (+4.2)
LaViDA-L base / StepRefineMathVista47.0 / 47.2 (+0.2)

VMCBench on LLaDA-V is 74.4, against WINO at 74.7. MMEval-Pro moves from 68.4 to 71.6, while ReMix falls to 63.5. The hallucination score barely travels: MMHalu goes from 3.35 to 3.50.

Stacked on accelerators, the speedups stay put. Fast-dLLM alone drops VStar from 55.5 to 52.4 at 8.45x; adding StepRefine returns it to 56.5 at 8.47x, and MMVP goes from 37.3 to 42.0, above the unaccelerated base of 40.7. dLLM-Cache moves VStar from 53.9 to 57.1, dKV-Cache from 54.5 to 56.0. Averaged over the three accelerators, VStar rises about 2.9 points and MMHalu about 1.6% after scaling by the 0-6 range. Tight budgets still help: at length 256 and only 64 denoising steps, MMHalu goes from 3.13 to 3.28.

Why it matters

Anyone already serving these MDLMs can attach the decoder without retraining, and without turning every step into a continuous state. The forward count stays on the original step budget, so Fast-dLLM, dLLM-Cache, dKV-Cache, and SparseD keep their speedups. The accuracy Fast-dLLM gives up on VStar and MMVP comes back, slightly above the unaccelerated baseline.

The target is narrow: discrete masked diffusion, at the few steps where uncertainty spikes and confidence stops ranking tokens. Errors baked into the training data are untouched, and autoregressive vision-language models are out of scope. The gain is incremental. On a 0-6 hallucination score, LLaDA-V picks up 0.15. What changes is where the budget is spent.

Limitations

The main text has no limitations section. The appendix case is a street scene where several rounds of soft refinement still miss the light color, the vehicle, and the time of day. The controller stops when the distribution settles. Settled is a different question from correct. If the first distribution at a spike already leans toward the wrong sense, more rounds only sharpen that error.

Gains are uneven. LaViDA-L picks up 0.2 on MathVista, 0.4 on the physics subset, and 0.4 on Geometry3K. The head-to-head with CoTA, ReMix, and WINO covers only LLaDA-V, and every backbone is 7B to 8B.

Tables 3 and 5 disagree with the prose on top-k margin remasking. Read in base-then-StepRefine order, MMHalu falls from 3.41 to 3.27 on LLaDA-V and from 3.39 to 3.25 on Dream-VL. The text says all three remasking rules improve. Random and low-confidence do improve, from 2.97 to 3.08 and from 3.35 to 3.50 on LLaDA-V. If the columns are not simply swapped, the gain depends on the commitment rule, and that dependence is left unexplained.

No table shows error bars or multiple seeds. The 80% overlap is a pooled observation. Refinement at spike steps does move scores, by a small amount, and there is no control that only delays commitment on those steps. Window 16, the mean-plus-sigma rule, and the 0.01 KL cutoff were chosen on these four models. The code is described as coming after acceptance.

Terms

Source

Related papers

All paper explainers