Entropy-aware distillation: letting student uncertainty decide when to imitate beats full token copying
valentina__py · x · 2026-10-12
A new paper, Learning What to Imitate: Entropy-Aware Distribution Mixing, shows that forcing a small LLM to copy every token from a stronger teacher can backfire.
- Method: the student's own uncertainty guides, token by token, how much to imitate the teacher via entropy-aware distribution mixing.
- Results: stronger generalization and less forgetting.
- Paper and code are open-sourced (team includes Imanol Schlag and Antoine Bosselut).
More from Research
- Live talk: MinerU-Popo paper on fixing OCR's cross-page context failures — HamelHusain · 2026-10-13
- OpenAI math repo's 'exact complex arithmetic' claim draws fire over arbitrary-precision assumption — jessi_cata · 2026-10-13
- PaTh Adds Recursive Latent Reasoning to Diffusion Models, Solving 92.5% of Hard Sudokus at 10M Params — jwvdm · 2026-10-13
- Open-source learnc auto-tunes C constants: 9.1% smaller JPEGs, +4.9 F1 on WebRTC VAD — keep_up_sharma · 2026-10-13
- Anthropic × Adaptyv protein design challenge 3 targets GLP1R, the Ozempic receptor — AllThingsApx · 2026-10-13
- Allen AI's TutorMoments Benchmark Finds LLM Tutors Over-Help Students — allen_ai · 2026-10-13