Look Only When Needed: Certificate-Guided Visual Routing for Efficient Diffusion VLMs
Donghyeon Cho, Joowon Kim, Seungho Shin, Eunho Yang
cs.CV, cs.AI, cs.LG
2026-10-27
Each step a cheap text-only forward verifies the last image-conditioned prediction; the image is rerun only on disagreement, keeping 99.6% of LLaDA-V's score at 0.35x FLOPs.
Diffusion vision-language models (DVLMs) such as LLaDA-V start from a fully masked response and run a full forward over image, prompt, and partial response at every denoising step. Bidirectional attention rules out the exact key-value caching that autoregressive models get for free, so every step recomputes everything. Visual tokens are 89-97% of the input, while a forward without them costs only 3-10% of the compute: most of DVLM decoding goes to reprocessing an image that never changes. Existing acceleration falls into three camps. Visual token pruning (RedVTP, ZOO-Prune) drops image tokens but loses that information permanently and still processes the rest every step. Intermediate-state caching (Fast-dLLM) reuses cross-step computation. Decoding schedules (KLASS) unmask more tokens per step. All three make image-conditioned forwards cheaper or rarer; none asks whether the current step needs the image at all.
The KAIST team measured first. Running LLaDA-V's trajectory with and without the image at every step: 83.4% of unmasked positions get the same top-1 token either way, and removing the image changes the unmasking decision at only 22.8% of steps on average, from 7.9% on MMBench to 43.8% on DocVQA. Image-dependent steps cluster early and do not recur periodically. A fixed schedule cannot track that, so image access has to adapt per input and per step.
LOWN (Look Only When Needed) splits apart two decisions that standard decoding bundles into a single forward, whether the step needs the image and how many tokens to unmask, and answers both with one training-free, threshold-free rule:
One certificate approximates two checks at once: the token does not need the image, and the anchor is still valid after the response state changed. Agreement alone suffices, with no absolute confidence threshold. Certificates do miss: a fresh FULL forward overturns 4.1-11.0% of tokens certified against a one-step-old anchor, and 2.2-4.7% on LOWN's own trajectories. The disagreement-first order compensates, since positions the image actually changed get unmasked first rather than deferred.
Ten benchmarks spanning math, science, charts, documents, and open-ended captioning, with every accelerated baseline given at least LOWN's FLOPs budget:
| Method (LLaDA-V, matched FLOPs) | Relative score | Relative decoding FLOPs |
| LOWN | 99.6% | 0.353x |
| ZOO-Prune | 94.2% | >= LOWN |
| RedVTP | 93.0% | >= LOWN |
| KLASS | 91.9% | >= LOWN |
LOWN scores highest on eight of ten benchmarks and beats the unaccelerated baseline on four; capping each at 100% gives 98.6% on average. One untuned rule covers all benchmarks, while the KLASS comparison uses per-benchmark thresholds. Wall-clock generation time drops 57.4% on average, up to 73.2%.
Compositions and transfer: with RedVTP at 50% retention it reaches 98.3% at 0.223x FLOPs and cuts RedVTP's own FLOPs by a further 57.6%; with the Fast-dLLM cache it cuts generation time by another 38.1-50.1% for a combined 5.6-7.0x speedup, though it scores below Fast-dLLM alone on all three tasks tested, by 5.0 points on LLaVA-Bench; the same rule transfers unchanged to LaViDa at 99.7% and 0.174x FLOPs; on the video model VidLaDA-8B it scores 43.3 on MLVU against the baseline's 41.8.
Ablations attribute most of the saving to visual routing alone (27-68% FLOPs reduction), with adaptive acceptance removing another 21-43%. Period 4 scores 4.0 points lower on MathVista at similar compute, and LOWN beats a random schedule matched in FULL-forward count by 4.9-13.0% on normalized quality. An oracle that consults the image only when it changes a decision costs 0.33x, against LOWN's measured 0.35x.
This is a decoding-layer change: no parameter updates, no attention modification, plug-and-play on LLaDA-family DVLMs. The bigger practical point is orthogonality with pruning. Pruning cuts the cost of each image-conditioned forward; LOWN cuts how often one runs; the savings multiply. Combined, a gentler pruning ratio wins on both quality and compute: on MathVista, RedVTP at 50% retention plus LOWN scores 54.9 at 0.195x FLOPs, better than RedVTP pruned to 30% retention with 51.7 at 0.354x.
There is also a structural advantage over pruning. Pruning deletes visual information before decoding; LOWN only skips the image at selected steps and can consult the full image later. The gap shows on fine-grained reading tasks: in the paper's qualitative example, RedVTP prunes the decisive evidence and answers fluently but wrongly, while LOWN answers correctly. For teams deploying diffusion VLMs, this is a model-free lever that stacks with the existing acceleration stack, and it establishes image-access scheduling as a new axis for DVLM efficiency.
The authors list three. Reused anchor tokens can differ from a fresh FULL prediction, more often as the anchor ages, up to 25.3% disagreement on DocVQA with an eight-step-old anchor (in practice 92-97% of tokens use an anchor from the preceding step, none older than five). Savings shrink on chart and document benchmarks where visual dependence persists. And total cost cannot be fixed before decoding, since the same agreement rule decides how many FULL forwards a response needs; only the pruning retention ratio gives indirect control.
Two gaps the paper does not cover. All experiments run batch size 1 with greedy decoding and classifier-free guidance off, so behavior under common deployment settings is untested. The Fast-dLLM composition also loses to Fast-dLLM alone on all three tasks, so stacking is not free. Transfer covers only LLaDA-family backbones (LLaDA-V, LaViDa, VidLaDA); other DVLM architectures remain untested.