Blogger Works Through Stanford CS31: Residuals Matter More Than Normalization
stanfordnlp · x · 2026-11-13
A blogger shares his reasoning while working through Stanford's CS31 deep learning quiz, which asks students to predict how removing transformer components affects validation loss. He found he had underestimated residual connections relative to normalization.
His updated mental model: some mechanisms determine what a model can compute (residuals), while others determine how reliably it learns that computation (normalization). He then analyzes RMSNorm via vector decomposition into length and direction: removing PreNorm while keeping residuals doesn't necessarily blow up hidden states — if the residual stream merely triples in length with unchanged direction, attention's relative feature patterns need not change drastically; RMSNorm is designed to damp exactly this scale sensitivity.
More from Research
- Incremental research is now riskier than going all in on big ideas — fkasummer · 2026-11-13
- MIT & Meta paper: ReHarness revises agent memory notes with later outcomes, beats baselines — rohanpaul_ai · 2026-11-13
- MASCOT (EMNLP 2026) trains multi-agent role-played dialogue, hitting 80.0 persona consistency — jiqizhixin · 2026-11-13
- Stanford's CS312 'Deep Learning Alchemy' opens quizzes and materials to the public — stanfordnlp · 2026-11-13
- Stanford Course Shares Data: AI Users Score Higher on Homework, Worse on Exams — stanfordnlp · 2026-11-13
- Cartwheel Claims Breakthrough in Hand-Object Kinematics Capture from Any Video — andrew_n_carr · 2026-11-13