Szegedy: a flaw in common training losses went unnoticed for over a decade
ChrSzegedy · x · 2026-10-27
Csaba Szegedy quotes Brandon Damos' findings on training-loss issues, amazed they went unnoticed for over a decade. The quoted thread notes that beyond soft cross-entropy, reverse KL and JSD — common in distillation — suffer similar issues, with fixes summarized. A notable observation about hidden flaws in distillation objectives.
More from Research
- UMAP arcing explained: one giant coordinate dominates neural network activation maps — leland_mcinnes · 2026-10-27
- NeurIPS benchmark: tree models still beat deep learning on 45 tabular datasets — tak3sh8 · 2026-10-27
- Kevin Bryan: maximizing raw accuracy is rarely the right way to train AI assistants — Afinetheorem · 2026-10-27
- Claude research model claims to refute Exponential Time Hypothesis with 14M-token autonomous run — scaling01 · 2026-10-27
- "ETH is false": CS community buzzing over claim that upends SAT exponential-time assumption — akbirthko · 2026-10-27
- RL environments essay: knowing how to operate software isn't knowing how to achieve goals — ShenRaphael · 2026-10-27