Kimi Delta Attention Brings Memory & Forget Gates: LSTMs Are Back
burny_tech · x · 2026-07-23
Developers noticed that Kimi's Delta Attention mechanism incorporates designs akin to memory and forget gates, prompting remarks that classic LSTM architectures are making a comeback in modern LLMs.
In the discussion, others pointed out that RNNs with looped transformers are also reviving, calling for the integration of forget gates into these modern structures and joking that Schmidhuber's influence is retrocausally everywhere.
More from Fun
- Dev builds city-sized zombie apocalypse sim in 100 days, ships on Steam — gdechichi · 2026-10-26
- 'Lamp Swarm': a desk of lamp-shaped robots shows what physical AI work looks like — dee_hw · 2026-10-26
- Dev-community banter: recommending a resource to every newcomer — itsOmSarraf_ · 2026-10-26
- Elon Musk's reply to a kid asking what's on Mars: "You get to decide" — yunta_tsai · 2026-10-26
- Prompt fills an empty room with everything your AI knows about you — jindong_wang92 · 2026-10-26
- The "claude is a lesbian" T-shirt is real and out in the wild — JasonBotterill · 2026-10-26