Why distillation gradients are noisy: single-token sampling breaks the expected-teacher estimate
jiank_uiuc · x · 2026-09-23
Technical follow-up from the same OPD paper author: the expected distillation gradient averages over possible next tokens, but in practice it's estimated from just one sampled token, making the update noisy even when the underlying teacher guidance is useful. The thread frames the problem in information geometry.
More from Research
- TissueMicroscopyBench: 49 evals to test AI agents on biological microscopy — kenbwork · 2026-11-12
- minpts-check: one-run predictor tells if your HDBSCAN clusters survive min_samples changes — leland_mcinnes · 2026-11-12
- LLM clinical ethics audit: near-deterministic decisions underweight patient autonomy — zakkohane · 2026-11-12
- Architecture study sparks monitoring concerns over a multi-hop planning spike — ChowdhuryNeil · 2026-11-12
- SGLang Megakernel Pushes Muse Spark to 1,109 tok/s, 2.1x Over Baseline — hsu_byron · 2026-11-12
- ArcGrav: a cross-model Agent Control Language to cut token overhead — ArchitecturalGravity · 2026-11-12