VAD: Attributing Visual Evidence for Target Reconstruction in Multimodal On-Policy Distillation
Kangning Zhang, Yixing Li, Shuai Shao, Qingyao Li, Zhengxi Lu, Zhiyuan Yao, Jianghao Lin, Wenxiang Jiao, Yuan Lu, Weiwen Liu, Weinan Zhang, Yong Yu
cs.CV, cs.CL
2026-07-31
VAD distills only the visual part of teacher corrections via counterfactual views; 4B/9B students hit 78.32/79.93 across six vision benchmarks, beating Gemini 3 and a 397B Qwen.
On-policy distillation (OPD) transfers fine-grained ability in vision-language models: the student generates its own decoding trajectory, and a teacher that has seen a "privileged view" (usually the relevant region cropped and zoomed) corrects it token by token. The teacher sees what the student cannot, and is meant to inject fine-grained visual skill.
The trouble is that every teacher correction is source-mixed. It bundles real visual evidence, language priors the model already had, and the teacher's own quirks. The authors ran a diagnostic: at the positions where the teacher corrects hardest, only 23.2% (4B) and 22.8% (9B) of corrections are actually explained by visual evidence. The other three-quarters is noise, and naively distilling it teaches the noise too.
Two prior lines fail to fix this. Vision-OPD matches the teacher's full distribution on the cropped image, visual and non-visual alike. VA-OPD weights only positions where evidence raises the probability of the correct token, so it misses refutation: when the evidence clearly shows the student is wrong, the right move is to push the mistaken token's probability down, and VA-OPD gives those positions zero weight.
VAD (Visual Attribution Distillation) is built on counterfactual attribution. Rather than asking where to distill and how hard, it first asks how much of a correction is actually supported by visual evidence.
The procedure has three steps.
First, at each student-generated position, the same fixed teacher looks at two images: one with the relevant visual evidence kept (the region cropped), one with it removed or degraded. The difference in centered log-probabilities is a signed vector ut, a proxy for the direction in which revealing evidence moves token probabilities. That is the visual-evidence direction.
Second, VAD projects the teacher's full correction rt onto this direction ut. The aligned component rt^vis counts as "explained by vision" and is kept; the residual rt^res is treated as language prior and teacher bias, and discarded. If rt and ut disagree in direction, the projection coefficient is clamped to zero and no visual shift is applied. This one-sided projection keeps only consistent support.
Third is target reconstruction. VAD does not copy the teacher's distribution; it starts from the student's own current distribution and changes only the token odds that the visual intervention can explain. The visual signal is split into a support branch (evidence raises the correct token) and a refutation branch (evidence lowers the wrong token), each with its own budget, and the support share is capped at τ+ to stop uncertain positive evidence from dominating. A weak teacher regularizer is added for stability.
The resulting target starts from the student and shifts only the odds the visual intervention explains.
Experiments use Qwen3.5-4B and 9B, 6,241 synthetic visual QA examples, and six fine-grained vision benchmarks (VStar, ZoomBench, HR-Bench 4K/8K, MME-RealWorld EN/CN), judged by GPT-OSS-120B.
| Method | 4B Avg6 | 9B Avg6 |
| Base Qwen3.5 | 70.85 | 73.86 |
| GRPO | 71.72 | 76.20 |
| VA-OPD | 74.92 | 76.67 |
| V-Zero | 75.42 | 77.13 |
| Vision-OPD | 75.92 | 76.88 |
| Decomposed OPD | 75.37 | 77.05 |
| VAD | 78.32 | 79.93 |
VAD tops both scales, beating the best scale-matched alternative by 2.40 at 4B and 2.80 at 9B. The parameter-efficiency story is sharper: the 4B checkpoint already exceeds Gemini 3 Flash (77.32), Gemini 3.1 Pro (78.04), and Qwen3.5-397B-A17B (76.49). At 9B, VAD wins five of six benchmarks; only ZoomBench favors VA-OPD by 0.35.
Does the attribution actually purify visual signal? A semantic analysis finds that visual attributes, objects, and A–D decisions make up 42.0% of the relative top-5 in rt^vis, against 26.7% in the raw correction rt and only 17.7% in the residual. Offline token effects move the right way: correct-token support rises from 7.52 to 7.89 points, wrong-token suppression from 6.29 to 6.61.
The cost is modest: 3.8% more step time than Vision-OPD at 4B (67.9 GPU-hours total) and 11.1% at 9B (96.3 GPU-hours). On four held-out tasks (MMVP, CV-Bench, MMStar, POPE), VAD is the only post-training method positive at both scales (+0.24 / +0.23); the others mostly negative-transfer.
The clean idea here: stop trusting the teacher blindly and ask, for each correction, what makes it a visual claim. The counterfactual costs two extra teacher forward passes (show the same image with and without the evidence, then subtract), and strips out most of the language prior and teacher bias from the supervision signal.
For anyone training VLMs, VAD is a drop-in replacement for Vision-OPD or VA-OPD. It costs single-digit-percent extra training time and buys a real edge on fine-grained vision (small text, fine detail, high resolution). Crucially it restores refutation: when evidence says the student is wrong, push the wrong answer down instead of leaving it alone.
Keep the gains in perspective. All six benchmarks are fine-grained vision; a general flagship like Gemini 3.1 Pro will win on other dimensions, and the 4B lead over it is a thin 0.28. VAD's claim is "more accurate at matched data and budget" and "small-model parameter efficiency," not a general win.
The authors name two. First, each position uses a single counterfactual pair (evidence present vs removed), giving one contrastive vector that may bias compositional evidence; multiple views or a learned directional basis could give a richer estimate. Second, the current projection yields semantic enrichment rather than identifiable separation: the attributed component can still carry non-visual teacher effects, and the residual stays mixed; cleaner attribution likely needs grounding-constrained decomposition.
One concern goes unverified: all six benchmarks sit in the narrow fine-grained-vision category. The paper reports nothing on general image-text understanding, long documents, or video, so the implicit "does not hurt general ability" claim rests on MMVP-level +0.2 numbers, which is thin. The training set is also only 6,241 synthetic examples, and robustness on real-world distribution is unexplored.