Kimi Delta Attention Brings Memory & Forget Gates: LSTMs Are Back
burny_tech · x · 2026-07-23
Developers noticed that Kimi's Delta Attention mechanism incorporates designs akin to memory and forget gates, prompting remarks that classic LSTM architectures are making a comeback in modern LLMs.
In the discussion, others pointed out that RNNs with looped transformers are also reviving, calling for the integration of forget gates into these modern structures and joking that Schmidhuber's influence is retrocausally everywhere.
More from Fun
- 'Even worrying about Hugging Face is a distraction': AI circle delights in mindfulness dodge — GarrisonLovely · 2026-09-11
- One-prompt 3D muscle anatomy site built on GPT-6 via WorkBuddy — lxfater · 2026-09-11
- Satirical startup claims Navier-Stokes singularity will cool GPUs, raising $1B seed — AIFlow_ML · 2026-09-11
- Nick Walton jokes about training a fly to solve a Rubik's cube — Al_Grigor · 2026-09-11
- After Planning Two Books with ChatGPT, He No Longer Feels the Need to Write Them — tinyfool · 2026-09-11
- After 9 Years, Bots Finally Show Up on a Blogger's Bounty Page with AI Text — gleech · 2026-09-11