Frame-by-frame real gradient descent: α=0.6 zigzags the valley, α=0.3 glides down

ZeroStateReflex · x · 2026-11-17

@mathelirium shared a gradient descent visualization showing how a machine "walks downhill" to fit the best line through data. Using linear regression with cost J(θ₀,θ₁)=1/(2m)Σ(h(xᵢ)−yᵢ)² and update rule θⱼ := θⱼ − (α/m)Σ(h(xᵢ)−yᵢ)xᵢ, two runs start from the same point on 14 data points: α=0.6 zigzags across the narrow valley of J while α=0.3 slides along it, residuals shrinking each step. Every frame is real gradient descent, with a bell ringing when the gradient falls to a thousandth of its starting value — a reminder of how something so simple powers nearly all of modern AI.

Original post →

More from Research

Research channel →