VCSD lets vision-language models self-distill from image-content contrast alone
burny_tech · x · 2026-07-27
Visual Contrastive Self-Distillation (VCSD) proposes a simpler way to do on-policy self-distillation for vision-language models by removing image content from the teacher signal without relying on external teacher models or privileged answers.
How it works
- For each student-generated response prefix, an EMA teacher produces two next-token distributions under the same prompt and prefix:
- one conditioned on the original image
- one conditioned on a content-erased control
- The token-level log-probability gap identifies tokens supported specifically by visual content.
- That contrast is used to sharpen the teacher’s original-image distribution and distill the resulting full-distribution target into the student.
Results
- Evaluated on ViRL39K.
- Outperforms matched OPSD across Qwen3-VL and Qwen3.5 models.
- On Qwen3-VL, the seven-benchmark aggregate improves from 62.27% → 67.04% at 2B, 71.30% → 73.16% at 4B, and 72.51% → 76.26% at 8B.
- Requires no external teacher, privileged answers, visual evidence signals, reasoning traces, or extra inference-time cost.
More from Research
- NUS builds a soft force sensor that drives actuators without electronics or power — CurieuxExplorer · 2026-07-27
- Chelsea Finn says robot RL is bottlenecked by physical rollout cost, not algorithms — ycombinator · 2026-07-27
- ICML 2026 oral paper replication scores stay middling after a stricter re-scoring — profjamesevans · 2026-07-27
- Long-running agents will need immutable event logs, this thread argues — sebpaquet · 2026-07-27
- Seed IQ navigates Doom II, prompting questions about benchmarks beyond ARC-AGI — Fit_Transition8824 · 2026-07-27
- Agentic Data Science in Practice: Agents Write Code but Answer Wrong Questions — hugobowne · 2026-07-27