Fewer Tokens, More Self-Teaching: On-Policy Self-Distillation for Extreme Visual Token Reduction
Junxian Li, Ruixuan Yang, Tianao Zhang, Tiange Xu, Weisheng Dong, Yulun Zhang
cs.CV, cs.CL, cs.LG
2026-09-26
On Qwen3.5-4B with 5% visual tokens, LT-OPD raises average retained performance from 68.6% to 82.3% by distilling a frozen full-token teacher on the student's own rollouts.
Cutting visual tokens is how you cheapen MLLM prefill. Training-free pruners such as FastV, DivPrune, CDPruner, and HiPrune still hold at 20% or 15% keep rates. At 5%, average retained performance across nine benchmarks falls to 59.0% to 77.6%. Choosing which tokens to keep does not rescue that budget.
Compression also changes the decode path. The student's next-token distribution drifts from the full-token model, then that output is fed back as context. EPIC, LLaVA-Mini, and LearnPruner still train on fixed dataset prefixes. A 5%-token student does not visit those prefixes at inference. It is short on pixels, and it is walking a different token path than the prefixes used in training.
LT-OPD splits one Qwen3.5 into a trainable low-token student and a frozen full-token copy as teacher. CDPruner, a parameter-free selector that mixes question relevance with conditional diversity, keeps fraction r of visual tokens for the student. The teacher sees every token.
Each update does three things. The student samples a response under the current budget; discrete sampling is stop-gradient. Teacher and student then read that same student prefix with different visual inputs and emit next-token distributions. Jensen-Shannon divergence matches the two: KL of each side to their mixture, averaged. The implementation evaluates the union of the student's top-100, the teacher's top-100, and the sampled token, with leftover mass in one tail bin.
On-policy means the training states come from the current student. The teacher never writes its own answer. After training, only the compressed student is served.
Early 5% rollouts are too broken for the teacher to correct, so the visual budget follows a curriculum: 25% for the first 14 updates, cosine decay to 5%, then a hold at 5% from update 101 to 175 (α=2, target β=5%). Data is LT-14K, 14,000 single-image questions from LLaVA-Instruct, OneThinker, PixMo, and Vision-OPD. The language model, vision encoder, and visual merger are all updated, 175 steps, 15.1 hours on eight A100s.
The headline metric is average retain ratio on Qwen3.5-4B at 5% visual tokens: each score divided by the full-token upper bound, then averaged over nine benchmarks.
| Method | Avg. retain at 5% |
| CDPruner, untrained | 68.6% |
| DivPrune (best training-free) | 77.6% |
| EPIC (trained baseline) | 73.7% |
| Same compression + SFT | 70.4% |
| Fixed-prefix JSD + curriculum | 77.6% |
| DAPO (best RL baseline) | 78.2% |
| LT-OPD without curriculum | 81.5% |
| LT-OPD | 82.3% |
LT-OPD wins 8 of 9 benchmarks at the 5% budget. V Bench is 74.9 versus DivPrune's 70.7; MME is 2,077.0 versus 1,915.4; OCRBench is 535 versus 477. The 5% student also beats every training-free method that keeps 10% of tokens, whose best retain ratio is 80.2%.
The same recipe transfers. Qwen3.5-9B goes from 71.1% to 81.4%, GLM-4.6V-9B from 73.8% to 86.3%, LLaVA-OV-1.5-4B from 71.8% to 84.1%. On multi-image MuirBench (52.0) and 16-frame MVBench (55.6), average retain is 78.3%, above CDPruner at 65.7% and EPIC at 69.8%.
KV cache falls from 26.59 MB to 3.93 MB (−85.2%). Prefill FLOPs fall from 6.002 T to 0.878 T (−85.4%), matching HiPrune at 0.146× full compute and undercutting EPIC's 0.200× by about 27%. End-to-end latency only moves from 1,331.5 ms to 1,293.7 ms (1.029×). Decode still dominates.
This is a training add-on for a model you already plan to prune, not a new pruner. Inference cost matches HiPrune; retained performance is 21.9 points above HiPrune's 60.4% at 5%. Teams that can spend about 15 hours on eight GPUs and mainly want cheaper prefill and KV can drop it onto a CDPruner pipeline.
The ablation is the useful part. Fixed-prefix JSD plus curriculum stops at 77.6%. SFT stops at 70.4%. GRPO, GSPO, and DAPO lift the untrained 68.6% to 75.8% to 78.2% and still lose to token-level distillation. A dense next-token teacher is a better signal than a sparse task reward when visual evidence is this thin.
Treat it as incremental. MMMU falls from 77.6 to 50.0. TextVQA falls from 81.2 to 56.0. Knowledge and reading are only partly recovered.
There is no dedicated limitations section. The gaps sit in the tables.
Training still needs the full-token teacher, so the serving win does not apply to the training job. The vision encoder is fully updated, heavier than adapter-only recipes. Curriculum knobs s1=14, s2=100, α=2 are empirical. Removing the final 5% plateau drops retain ratio from 82.3% to 79.5%; those 75 hold steps matter, and the paper does not say why the hold has to be that long.
Wall-clock barely moves. An 85% prefill FLOPs cut buys 1.029× end-to-end, which will not change decode-bound online serving. Main tables are single-image. Multi-image and video appear only in an appendix, and MVBench uses 16 uniformly sampled frames. The claim that LT-OPD does not depend on a best selector is untested: training is wired to CDPruner, with no HiPrune or DivPrune cross.
The objective is not uniquely best either. Reverse KL reaches 82.8% average retain, a hair above JSD's 82.3%. JSD wins on perception-heavy sets. OCR-heavy users may want KL.