Stanford CS312 quiz: what happens if you pin the residual stream's RMS mid-training?
tatsu_hashimoto · x · 2026-11-13
A prediction quiz from Stanford's CS312 "Deep Learning Alchemy" tests intuition on how residual connections and layer norms interact during training. Setup: train a 35M-parameter, 8-layer Transformer LM for 9,375 updates, then fork at step 4,688 into two runs. Control continues normally; in the modified run, after every attention/MLP residual addition, activations are rescaled to match the RMS measured at the fork point (targets from 16 validation sequences, then frozen — post-norm-like scaling with no mean-centering or learned gain).
Answer options range from "improves by >0.02" to "worsens by >0.15 or divergence," forcing you to reason about how the residual stream's scale drifts during training and whether freezing it breaks downstream layer adaptation.
Related event: Freezing Residual Stream RMS Mid-Training Barely Hurts Loss(2 posts)→
More from Research
- Speculative Self-Distillation cuts training tokens 38–69% with no accuracy loss — Azaliamirh · 2026-11-13
- tszzl: local minima are rare in high-dimensional spaces — the greatest whitepill — JensHonack · 2026-11-13
- Chronicle Bio announces first clinical trial with Prolong, using AI to match patients for ME/CFS drug — thekaransinghal · 2026-11-13
- Salesforce's A²-Bench: 500 scenarios expose 20 failure modes in multi-agent teamwork — mohitban47 · 2026-11-13
- Matrix optimizers likely within 5-10% CE of each other, researcher argues — _arohan_ · 2026-11-13
- Princeton/Stanford study: optimizer rankings reverse as batch size changes — _arohan_ · 2026-11-13