Are logit-KL and hidden-state methods still 'distillation'? A naming debate
cephaloform · x · 2026-10-15
The author argues the gap between SFT on traces and "SFT on traces + full logit KL + hidden state translation MSE" is too large to call them both "distillation," challenging how loosely the term is used for post-training methods.
More from Research
- Gil Kalai presents 'the most outrageous conjecture' on proving primes in sparse sets — _onionesque · 2026-10-15
- 83,017-vacancy RCT: human recruiters add no hires once algorithmic screening exists — danielrock · 2026-10-15
- RAESR Does Super-Resolution in Frozen DINOv3 Latent Space at 37ms per 512px Image — RTP-LLM · 2026-10-15
- BigCell Generates Coherent Gigapixel Whole-Slide Pathology Images, Boosting Few-Shot Classification 18% — StonyBrook-CVLab · 2026-10-15
- NavGPT-3 Reaches Human-Level Navigation: 90.43 SR on RxR-CE vs Human 90.4 — Metacognition-AI · 2026-10-15
- ByteDance Seed's Cortex Distills Context into Parametric Memory, Keeping 88.9% Gains — ByteDance-Seed · 2026-10-15