Freezing Residual Stream RMS Mid-Training Barely Hurts Loss

Tatsu Hashimoto shared a Stanford CS312 quiz experiment showing that freezing the residual stream RMS halfway through training a 35M-parameter Transformer raises validation loss by only 0.0043.

2026-11-13 ~ 2026-11-13 · 2 related posts

• Stanford CS312 quiz: what happens if you pin the residual stream's RMS mid-training? — tatsu_hashimoto · 2026-11-13
• Freezing residual-stream RMS mid-training changes final loss by only +0.0043 — tatsu_hashimoto · 2026-11-13