VRSD adds verbal regularization to self-distillation, lifting Pass@1 by 3.9 points
KingsCollegeLondon · hf · 2026-10-14
Researchers from King's College London and Tencent YouTu Lab propose VRSD (self-distillation with verbal regularization), an improvement over on-policy self-distillation (OPSD) for training LLMs.
- Problem: in OPSD, the teacher conditioned on privileged information (e.g., verified solutions) biases supervision toward shortcut reasoning with little verification, accompanied by a sharp drop in output entropy
- Method: system prompts regulate how the teacher uses privileged information, encouraging rethinking and exploration; theoretically, prompting is modeled as attenuating the influence of privileged information, with derived entropy-preservation conditions
- Results: VRSD beats OPSD on Pass@1 in 11 of 12 model-task settings across 3 models and 4 tasks, by 3.9 points on average, while generally maintaining higher token entropy
More from Research
- FlashVSR+ runs untiled 4K video super-resolution on one RTX 4090 with 2x speedup — MMLab-CUHK · 2026-10-14
- Sber's GreenWAM world foundation model spans 20 robot families as both predictor and policy — SberRoboticsCenter · 2026-10-14
- Tencent's LifeSearch benchmark: best deep research agent scores just 63.4% on real-life tasks — Tencent-Hunyuan · 2026-10-14
- Science: Menopause may explain why women face higher dementia risk than men — AnnaCiaunica · 2026-10-14
- When will AI start delivering breakthroughs in physics like it has in math? — burny_tech · 2026-10-14
- Photonic Training of a 1B GPT-2 Without Backprop Lands in PNAS — IgorCarron · 2026-10-14