Yang Zhilin quotes Ilya: 'an LSTM is a ResNet rotated 90 degrees'
SchmidhuberAI · x · 2026-10-15
Schmidhuber highlights that Moonshot AI founder Yang Zhilin quoted Ilya Sutskever's line — "an LSTM is a ResNet rotated 90 degrees" (from Ilya's NeurIPS 2024 talk). Rotating an LSTM yields residuals: residual connections can be viewed as a simplified LSTM unrolled over depth, providing only a narrow information highway through fixed addition; attention rotated 90° is then a natural generalization of residuals, letting each layer retrieve broader, input-dependent representations.
Schmidhuber uses the moment to re-assert his priority timeline: Hochreiter's 1991 recurrent residual connections (weight 1.0) solved vanishing gradients, predating the 21st century's most-cited deep residual learning paper.
Related event: Yang Quotes Ilya's 'LSTM Is a Rotated ResNet'; Schmidhuber Cites History(2 posts)→
More from Fun
- Viral meme nails AI's irony: it can write symphonies but can't do the dishes — briannekimmel · 2026-10-16
- Man with cerebral palsy uses HeyGen avatar to tell his life story in his own voice — HeyGen · 2026-10-16
- Researcher jokes that AI has officially achieved dad jokes internally — keyanzhang · 2026-10-16
- 'Clawds are grown on trees': community Claude meme gets an Opus-made animation — edwinarbus · 2026-10-16
- mitsuhiko's AI agent solved multiplayer with PeerJS before he could explain it — mitsuhiko · 2026-10-16
- Proud dad shares 12-year-old's ChatGPT account: kid built his own state-memorizing game — OpenAIDevs · 2026-10-16