Xiaohongshu's D³-MOPD dynamic distillation closes 97% of teacher-student gap, 3x faster
小红书技术REDtech · wechat · 2026-09-07
Xiaohongshu's AllSpark team proposes D³-MOPD to fix a flaw in multi-teacher online distillation: fixed data ratios waste compute on already-converged domains. The method reuses the reverse-KL signal training already computes to dynamically reschedule data mixing, scoring each domain by "how far from teacher" × "how fast improving", with a minimum-ratio floor and temperature control. Architecture is lightweight (async watcher + hierarchical sampler, zero changes to the main training loop). On Qwen3.6-35B-A3B with four teachers across math/code/instruction/tool-use, it closes 97% of the teacher-student gap (vs 63% for vanilla MOPD), beats teachers on 3 of 7 benchmarks, and converges 3x faster (47 vs 143 steps to 61.4 avg). Paper: arxiv.org/abs/2608.24987
More from Research
- JEVfire open-sourced: Qwen 0.8B clears Super Mario in-browser at 71ms per action — ricklamers · 2026-09-22
- RegVGGT: training-free token regulation keeps only 1% of tokens per frame for streaming 3D reconstruction — zhenjun_zhao · 2026-09-22
- D3GS: depth, DINO and diffusion co-guided 3D Gaussian Splatting for sparse-view reconstruction — zhenjun_zhao · 2026-09-22
- VGGT-Prime: compute-adaptive mixture-of-heads slashes redundancy in visual geometry transformers — zhenjun_zhao · 2026-09-22
- Elevator-VIGS: Gaussian Splatting SLAM that keeps tracking through elevator rides, zero-shot via VLM — zhenjun_zhao · 2026-09-22
- BayesianGS-SLAM: Uncertainty-Aware 3DGS SLAM Uses Bayesian Uncertainty Across Mapping, Tracking and Keyframes — zhenjun_zhao · 2026-09-22