KAN author's new paper: loss spikes are predictable, quadratic loss picture wrong
ZimingLiu11 · x · 2026-11-01
Ziming Liu (KAN co-author) shares a new paper on predicting LLM training loss spikes: "spiky" is just one phase in the (β₁, β₂) plane, sandwiched between two stable phases, and the traditional "quadratic loss" picture is seriously wrong — a core-wall-rollover loss landscape is observed instead. Thread with explainer video and paper link.
More from Research
- BioCompute opens its remotely controlled wet lab, pitching 'Lovable for biology' — vaibhavbetter · 2026-11-04
- Legora CTO hand-writes 100k-param model weights with Fable, one-shot beats training — dair_ai · 2026-11-04
- 20-40% of alignment data is noisy: a loss-reduction scoring framework to fix it — burkov · 2026-11-04
- Scientists image quantum entangled photons in real time, and the result looks like a taijitu — CurieuxExplorer · 2026-11-04
- kalomaze: crude diffusion autoencoders may lose low-frequency info at the latent stage — kalomaze · 2026-11-04
- Structured captions boost text-to-music alignment, new Music DSG metric beats CLAP — affige_yang · 2026-11-04