SCOPD: Sparse-Context On-Policy Self-Distillation for Efficient Vision-Language Models
Ahmadreza Jeddi, Enming Zhang, Jasper Gerigk, Hakki Karaimer, Mozhgan Nasr Azadani, Jiayun Luo, Minh Ngoc Le, Gholamali Aminian, Hugo Buurmeijer, Yongchao Chen, Leonid Sigal, Igor Gilitschenski, Konstantinos G. Derpanis, Marco Pavone, Babak Taati
cs.CV
2026-09-28
At 10% VisionZip tokens, Pass@64 recovers 79.6% vs 53.2% greedy. SCOPD+ distills a sparse student from a full-context teacher, lifting 13-benchmark retention from 86.37% to 92.43%.
Reasoning VLMs encode images and videos as hundreds or thousands of visual tokens, so language-model prefill dominates inference cost. Training-free token pruning can cut that context down, but scores fall hard under aggressive compression. The usual account is that task-relevant visual evidence was deleted and cannot come back.
A fixed-context Pass@K study says that account is incomplete. On 500 free-form numerical items the unpruned model already solves greedily, VisionZip keeps 10% of tokens once, then every sample reuses that same sparse visual representation. Greedy success falls to 53.2%, yet Pass@64 climbs to 79.6%, and 74.20% of items have at least four valid traces in 64 draws. A language-only control, image removed, only moves from 2.8% to 4.2%. Because the retained tokens never change, recovery is not a luckier prune. The evidence is still in the sparse context; the language model fails to use it reliably. The paper names this the representation-utilization gap.
The response is to adapt the language model to the sparse visual context it will see at deployment, and leave the pruner as a frozen, training-free operator.
SCOPD (Sparse-Context On-Policy Self-Distillation) has the student roll out think / answer traces from pruned visual tokens, while a teacher that sees the full visual sequence scores the same prefixes with token-level forward KL. Autoregressive states depend on the student's own past tokens, so the teacher has to supervise those on-policy prefixes. The teacher is an EMA of the student (decay 0.9999). No ground-truth answers, no architecture changes, no extra inference compute.
Large teacher-student KL is a noisy proxy for missing vision. Synonyms, capitalization, and newlines can blow up KL while extra visual tokens barely move the next-token distribution. SCOPD+ therefore runs a small visual-budget intervention: the student generates at budget b, then the same prefixes are scored at b+δ (default δ=1%). Jensen-Shannon divergence between those two distributions ranks positions, and only the top ρ=10% are distilled. JSD is symmetric and bounded, which makes a small extra visual budget easier to compare across tokens. Backprop then touches only a slice of the response.
Training stays small: Qwen2.5-VL-7B-Instruct, LLM-only LoRA (r=16, α=32), vision encoder and projector frozen, about 10K LLaVA-CoT image-question pairs, VisionZip at 10% during training. Baselines on the same sparse context include SFT on reference traces, EPIC's progressive consistency distillation, and GRPO with outcome rewards.
Avg13 is the mean of 13 image benchmarks, each normalized to unpruned Vanilla. Image inputs are capped at 4096 visual tokens; videos use 32 uniformly sampled frames, up to 12,288 tokens. Final answers are read from the answer field; Qwen3.6-27B is used only when string matching is ambiguous.
| Method | Avg13 at 10% tokens | Avg13 at 20% tokens |
| Vanilla | 86.37% | 93.42% |
| SFT | 88.82% | 95.22% |
| EPIC | 86.45% | 94.71% |
| GRPO | 85.73% | 93.95% |
| SCOPD | 90.49% | 96.80% |
| SCOPD+ | 92.43% | 97.25% |
With a full visual context, SCOPD and SCOPD+ sit at 100.67% and 100.02%. The gain grows as the visual budget shrinks, which fits adaptation to sparsity rather than generic post-training strength.
At 10%, SCOPD+ beats dense SCOPD on 8 of 13 benchmarks and ties on two. Against SCOPD, LogicVista moves from 34.68 to 38.26, MathVerse from 30.46 to 33.38, VisOnlyQA from 42.09 to 43.65, HRBench-4K from 63.75 to 66.00. Against unadapted Vanilla, MathVerse goes from 26.90 to 33.38 (unpruned 42.39) and VisOnlyQA from 40.35 to 43.65 (unpruned 44.70). SFT reaches 68.00 on HRBench-4K, above SCOPD+'s 66.00, but its Avg13 is still 88.82%. GRPO falls to 85.73%, below Vanilla.
Which positions get distilled matters. With the same 10% of response tokens, random selection scores 93.56 Avg6, top teacher-student KL 94.51, TIP 94.21, least-sensitive positions 89.13, visual sensitivity 95.25, versus 94.44 for dense SCOPD.
Trained only on VisionZip, SCOPD+ still helps other pruners with no extra adaptation: DivPrune 83.47% to 91.23%, random pruning 82.06% to 88.85%, FastV (pruning inside the LLM) 81.47% to 86.03%. On Qwen3-VL-4B at 10% tokens, Avg6 goes from 75.29% to 81.6% (SCOPD) and 82.92% (SCOPD+). Image-only post-training, no video fine-tune, lifts five video benchmarks from 90.88% to 95.70% / 96.60%. At 5% retention on eight image sets, Vanilla holds 75.99%, SCOPD 83.63%, SCOPD+ 83.66%.
For anyone shipping a reasoning VLM, this is a cheap post-training path: keep an off-the-shelf training-free pruner, adapt only LLM LoRA, add nothing at inference, and skip reasoning-trace labels. SCOPD+ costs one extra sparse forward at train time: 21.4% more theoretical compute than SCOPD, 1.9% more wall-clock, essentially unchanged peak memory.
The diagnostic travels even if the method does not. A drop after pruning is not automatic proof that the evidence is gone. If Pass@K still climbs on a frozen sparse context, some of the failures are utilization. Training can then target using what survived, rather than only swapping in a smarter token selector.
The improvement is incremental. At 10% tokens SCOPD+ still sits at 92.43% of the unpruned model, and MathVerse remains well below 42.39. Adaptation can recover use. It cannot restore details that pruning actually deleted.
The authors trained mainly with VisionZip and fixed retention budgets. Transfer to other operators is shown, but dynamic pruning, layer-adaptive sparsity, and jointly learned compression are left open. Evaluation is short-form image QA plus 32-frame video. Longer video, embodied, and agent settings are untested, and those may lose surviving evidence over time.
The Pass@K set is 500 numerical items already solved greedily without pruning, not an official leaderboard. 79.6% is recoverability of previously solved cases, not overall strength after pruning. A GPT-6-Astra-Max judge also filters whether the sampled reasoning is visually grounded; judge bias lands in that number.
Most runs use Qwen2.5-VL-7B. Qwen3-VL is only a 4B model on a six-benchmark slice. Using the current student weights as teacher collapses Avg13 to 12.35, so the EMA teacher is required. At 5% tokens SCOPD and SCOPD+ are essentially tied, which is what follows once information is actually gone. Main scores are greedy; the diagnostic samples at temperature 0.7.