Will LSTM Architectures Crush Transformers in 2-3 Years?
heghbalz · x · 2026-07-31
Francois Chollet and others discussed a radical architectural evolution: LSTM-based architectures could replace current Transformers within 2-3 years.
The core argument is that relying on KV Caches is dumb and clumsy. Future LSTMs should utilize a constant hidden state integrated with large external memory banks via tools like Mem0 or RAG. This mimics humans using notebooks rather than forcing the model to memorize all facts (like celebrity trivia or city populations) inside its weights.
Ideally, the model would only learn functions to manipulate, write, and retrieve facts. This lightweight model with a fixed hidden state could reside entirely in SRAM, while external "memories" sit in CPU RAM, massively boosting efficiency.
More from Research
- Top ML Conferences Receive 100k Papers Yearly, But Can We Name 10 Greats? — andrewgwils · 2026-07-31
- Study: Over-reliance on Coding Agents Hurts Code Comprehension — omarsar0 · 2026-07-31
- Open Models Accelerate Specialization as the Post-Training Stack Matures — bigdata · 2026-07-31
- Microsoft Open-Sources TRELLIS: Turns Any Image into 3D Assets in 3 Seconds — OwariDa · 2026-07-31
- ECCV 2026 Announces Call for Papers for Multimodal Digital Agents Workshop — LINJIEFUN · 2026-07-31
- From Basics to Agentic RL: A Deep Dive into LLM Tool Use — natolambert · 2026-07-31