Freezing residual-stream RMS mid-training changes final loss by only +0.0043
tatsu_hashimoto · x · 2026-11-13
A CS312-style experiment from Tatsu Hashimoto: train a 35M-param, 8-layer Transformer LM for 9,375 updates, then pin the post-residual RMS norms halfway through (update 4,688).
- Final validation loss rises by only +0.0043 vs. control — almost no change.
- Pinning from initialization is disastrous instead: default loss 2.9264 vs. 6.8746 when norms are frozen at init.
- Why: the residual stream's scale changes substantially in early training; by halfway, the activation RMS profile across depth is stable, so freezing it barely matters.
Echoes early post-norm vs pre-norm findings about how residual connections and LayerNorm interact during training.
Related event: Freezing Residual Stream RMS Mid-Training Barely Hurts Loss(2 posts)→
More from Research
- OpenKB: open-source LLM knowledge base hits 4.9k stars with vectorless reasoning-based retrieval — tom_doerr · 2026-11-13
- PhysCaP: NTU and NVIDIA's agents measure mass and stiffness with no extra sensors — shaohua0116 · 2026-11-13
- METR time-horizon fit called too pessimistic, redone — with a superexponential meme chart — lxrjl · 2026-11-13
- Meta's Papailiopoulos: Training's Future Is Agents That Compile Intent Into Fully Bespoke FLOPs — DimitrisPapail · 2026-11-13
- Further fine-tuning CLIP with contrastive losses often fails on out-of-distribution data — wightmanr · 2026-11-13
- Interaction-as-a-Judge beats code rewards: +6.7pp win rate, 35.7% less code bloat — arena · 2026-11-13