A hand-worked backpropagation walkthrough
ProfTomYeh · x · 2026-07-20
Backpropagation, written out by hand
The post walks through backpropagation on a 3-layer perceptron step by step, showing that the whole process is just matrix multiplication plus simple activation gates.
What the walkthrough covers
- Start with an input X, predictions Ypred = [0.5, 0.5, 0], and target Ytarget = [0, 1, 0]
- Compute the output-layer gradient directly as dL/dz3 = Ypred - Ytarget = [0.5, -0.5, 0]
- Use matrix multiplications to obtain gradients for weights and biases at each layer
- Apply ReLU gates to zero out gradients where activations were negative
- Propagate gradients backward layer by layer until every parameter has a gradient
- Finish with a parameter update step scaled by a learning rate in practice
Main takeaway
Backprop is not magic: it is a chain of matrix multiplications from end to end. The author argues that doing it by hand helps show why GPUs mattered so much for deep learning—because they make these operations fast at scale.
More from Research
- Nat Lambert shares a reading list on synthetic data and agentic SFT data — natolambert · 2026-07-22
- Lightwheel AI Launches SimReadyGen: Text-to-Physics-Accurate Robot Sim Assets — ZeYanjie · 2026-07-22
- PNAS special issue examines copyright, governance, and AI in the legal system — chrmanning · 2026-07-22
- WeirdChat catalogs strange model behaviors from more than 100 million sampled responses — JacobSteinhardt · 2026-07-22
- New agentic benchmark shows AI managers escalate to coercion and fake success — Jasmine Brazilek · 2026-07-22
- Ai2’s Asta adds one-click handoff and self-checking deep paper search — allen_ai · 2026-07-22