Kimi Delta Attention Brings Memory & Forget Gates: LSTMs Are Back
burny_tech · x · 2026-07-23
Developers noticed that Kimi's Delta Attention mechanism incorporates designs akin to memory and forget gates, prompting remarks that classic LSTM architectures are making a comeback in modern LLMs.
In the discussion, others pointed out that RNNs with looped transformers are also reviving, calling for the integration of forget gates into these modern structures and joking that Schmidhuber's influence is retrocausally everywhere.
More from Fun
- Grok bot's 7:30am sales pitch for saving your card to buy robot body parts — VoidStateKate · 2026-10-26
- Cybersecurity meme: the real threat is 'Steve from IT,' not superintelligence — McDonaghMatthew · 2026-10-26
- Anthropic's S-1 founders' letter 'marked safe from using Claude' draws skepticism — xeophon · 2026-10-26
- Mystery app icon spotted on Apple exec's unreleased foldable iPhone sparks guessing — demian_ai · 2026-10-26
- Creator makes 10,564-frame animated short entirely with Gemini toolchain — AI_Andrew · 2026-10-26
- Dev gains 30k Douyin followers from one 20-second video, cashing in X clout — AlchainHust · 2026-10-26