A step-by-step hand derivation shows how residual connections power deep nets
ProfTomYeh · x · 2026-07-28
A hand-calculated walkthrough of ResNet and transformer residuals
This post walks through the logic of residual connections by hand, using small matrices and step-by-step examples.
It starts with the classic ResNet idea from Deep Residual Learning for Image Recognition: the identity shortcut lets a block compute F(x) + x, which helps avoid vanishing and exploding gradients and made very deep networks practical.
The walkthrough then applies the same idea inside a transformer:
- in attention, the residual path combines the attention output with its input
- in the feed-forward block, the identity path again preserves and adds the original signal
The post’s takeaway is simple: a small “add” operation is what made much deeper architectures work in practice.
More from Research
- A ripgrep crash investigation ends up finding a kernel TLB-shootdown race — BenBajarin · 2026-07-28
- Curriculum learning can keep reinforcement-learning agents from exploiting random luck — ShawnHymel · 2026-07-28
- OpenAI and Anthropic may soon compete against their customers’ own data loops — bigdata · 2026-07-28
- SWE-rebench expands to five languages and adds local Qwen models — Fabulous_Pollution10 · 2026-07-28
- University of Basel is hiring PhD and postdoc researchers for LLM training — PMinervini · 2026-07-28
- Rednet could reuse much of Bittensor’s stack but ditch global consensus — sull · 2026-07-28