Softmax bug #2: distillation gradients are non-zero even when student equals teacher
jon_barron · x · 2026-10-24
Jon Barron details the second bug, "Softmax Cancellation": when output logits are very close to soft target logits (e.g. distillation with student initialized as teacher), current implementations produce non-zero gradients even when student == teacher. Fixing cross-entropy removes slingshot spikes and speeds up grokking; weight decay and Adam's eps (default 1e-8, right past the fp32 bug threshold) conspire to bound logit growth — possibly why small eps is considered dangerous.
More from Research
- Stanford-led paper traces 'bigger is better' to data-induced competition for neurons — niloofar_mire · 2026-10-24
- Frequency drives conventionalization: higher-exposure communities conventionalize more words — abenitezburraco · 2026-10-24
- GeoWind2Plan: seconds-scale 3D urban wind prediction for energy-efficient drones — AnimaAnandkumar · 2026-10-24
- Flowing water and lava come to 3D Gaussian Splatting scenes — jonstephens85 · 2026-10-24
- Immorta Bio's AI-designed 9-amino-acid peptide SenoMune-0257 targets senescent cells — DavidmComfort · 2026-10-24
- France rebuilt its tax code as MLang, a formally verified open-source compiler — fkasummer · 2026-10-24