Grokking isn't magic: weight decay contracting spatial oscillations explains generalization
burny_tech · x · 2026-10-01
A technical thread argues that grokking—memorizing the training set long before generalizing—is not explained by flat minima. The author proposes that networks fit training points on a fast timescale, while out-of-sample generalization is a separate, slower process: weight decay gradually contracting spatial oscillations across the input domain. The thread walks through the math behind this claim.
More from Research
- PKU, Tsinghua, Alibaba open-source SparkDiffusion: 265x faster AI video on one RTX 5090 — lmoroney · 2026-10-01
- AgentTell Benchmark: Browser Agents Leak Secrets via Behavior in 61% of Sessions — kagnlp · 2026-10-01
- Qwen3.8-Flash-Next Cut 44% via REAP Hits 70% on Terminal-Bench 2.1 — rmonsurate · 2026-10-01
- NVIDIA's LongLive-Plug distills video generation into plug-and-play LoRAs for 54 models — _akhaliq · 2026-10-01
- Omni-IO Skills: open-source harness lifts agent multimodal support rates from under 40% to 100% — _akhaliq · 2026-10-01
- LEGO-Anything: Coding Agents Rebuild Single Images into Editable 3D Scenes — _akhaliq · 2026-10-01