ResNet by hand: a 10-step walkthrough showing how one identity matrix enabled thousand-layer networks
ProfTomYeh · x · 2026-09-12
Prof Tom Yeh walks through ResNet's residual connection in 10 hand-computed matrix steps, explaining why He Kaiming's CVPR 2016 paper is among the most cited in deep learning: it fixed exploding/vanishing gradients with an identity matrix.
- Linear + ReLU: multiply inputs by weights, add bias, apply ReLU to get F(X).
- Concatenation trick: stack an identity matrix beside the layer weights and inputs beneath features — the identity is the residual, the lines are skip connections.
- One multiplication computes F(X)+X: the identity carries X through while weights transform it.
- Same trick in transformers: two stacked identities in attention (combining positions across columns), repeated in the feed-forward layer (combining features across rows).
Takeaway: one simple "add" is what made truly deep networks possible.
More from AGI Musings
- AI takeover scenarios skip a step: a willing-but-incapable model would fail first, argues Purtell — JoshPurtell · 2026-09-12
- Gary Marcus pushes back on prediction that Anthropic defectors will return within 6 months — GaryMarcus · 2026-09-12
- OpenAI Wasn't 'Out of Control' — It Was a Calculated Trade-off, Argues User — kchonyc · 2026-09-12
- EA grant platform hires FTX's Caroline Ellison, drawing fire over alignment judgment — zetalyrae · 2026-09-12
- RL debate: REINFORCE is both policy gradient descent and a synthetic data method — jessi_cata · 2026-09-12
- OpenAI was never 'out of control' — it could always shut down its data centers, argues critic — kchonyc · 2026-09-12