Kimi Delta Attention Brings Memory & Forget Gates: LSTMs Are Back
burny_tech · x · 2026-07-23
Developers noticed that Kimi's Delta Attention mechanism incorporates designs akin to memory and forget gates, prompting remarks that classic LSTM architectures are making a comeback in modern LLMs.
In the discussion, others pointed out that RNNs with looped transformers are also reviving, calling for the integration of forget gates into these modern structures and joking that Schmidhuber's influence is retrocausally everywhere.
More from Fun
- British-born monk becomes first foreigner to reach Thai Buddhism’s second-highest rank — StewartalsopIII · 2026-07-23
- Amazon AGI Lays Off Pretraining Data Engineer: "I Got Filtered Out" — yuwen_lu_ · 2026-07-23
- A meme says a theory not disproven by ChatGPT may be a bad sign — silver__tsuki · 2026-07-23
- OpenAI discussion quote about rapid RSI turns into an alignment meme — zetalyrae · 2026-07-23
- A therapy startup almost registered a domain no one could unsee — daluoseo · 2026-07-23
- A meme turns “quantum AI will achieve ASI” into the joke it sounds like — GlenBradley · 2026-07-23