Fixing cross-entropy removes grokking spikes; Adam's default eps=1e-8 sits right past the bug
jon_barron · x · 2026-10-24
Follow-up from Jon Barron: fixing cross-entropy removes slingshot spikes and speeds up grokking. Loss curves show standard CE oscillating wildly during grokking while the fixed version converges smoothly.
He also notes that weight decay and Adam's eps conspire to cap logit growth, and the default eps=1e-8 happens to sit just past where the bug triggers in fp32 — possibly why small eps is considered dangerous. Charts show Adam step size collapsing to zero once the logit margin exceeds the precision threshold.
More from Research
- Frequency drives conventionalization: higher-exposure communities conventionalize more words — abenitezburraco · 2026-10-24
- GeoWind2Plan: seconds-scale 3D urban wind prediction for energy-efficient drones — AnimaAnandkumar · 2026-10-24
- Flowing water and lava come to 3D Gaussian Splatting scenes — jonstephens85 · 2026-10-24
- Immorta Bio's AI-designed 9-amino-acid peptide SenoMune-0257 targets senescent cells — DavidmComfort · 2026-10-24
- France rebuilt its tax code as MLang, a formally verified open-source compiler — fkasummer · 2026-10-24
- Running RFD4-Proteina protein design model at thinking_high on consumer cloud GPUs — AllThingsApx · 2026-10-24