LSTM by hand: a 15-step walkthrough of every gate and memory cell
ProfTomYeh · x · 2026-09-17
Prof. Tom Yeh's latest "AI by Hand" post walks through computing an LSTM cell entirely on paper over a sequence of three inputs, in 15 steps.
- Context: Introduced by Hochreiter & Schmidhuber in 1997, LSTMs were the best approach for long sequences until Transformers took over; with attention struggling to scale to hundreds of thousands of tokens, recurrence (e.g. Mamba) is back in fashion.
- The walkthrough: Given X1–X3 and four weight matrices (forget, input, candidate, output), initialize h0 and C0 → apply linear transforms to the stacked input, previous hidden state and a 1 → squash three results with sigmoid to get the gates → update the memory cell (old memory × forget gate + candidate × input gate) → apply tanh and multiply by the output gate for the new hidden state → repeat for X2 and X3.
Finish it and you'll genuinely understand what every gate and memory cell does in an LSTM.
More from Research
- Research Culture Broken: Corrections, Replication, Negative Results All Unrewarded — tak3sh8 · 2026-09-17
- Keller Jordan: claiming AI could ruin math implies mathematicians can't self-govern — kellerjordan0 · 2026-09-17
- Dev Says Frontier Lab's Jev Architecture Mirrors His Open-Source Work From a Year Ago — Nandakishor_ml · 2026-09-17
- Nvidia's Agora: 13 LLM Agents Run 12 Days Unsupervised via Git-Based Shared Memory — nvidia · 2026-09-17
- HypoEvolve: Genetic Algorithms Orchestrate Multi-Agent LLMs for Cancer Drug Repurposing — Jieyuan Liu · 2026-09-17
- OpenAI Foundation funds "Public Data for Health" to create biology datasets for AI — DrMorganLevine · 2026-09-17