Meta Introduces Sparse Delta Memory to Enhance RNN

Meta AI introduced Sparse Delta Memory (SDM), combining GatedDeltaNet updates with Product Key sparsity to enhance linear RNNs. This method expands recurrent state capacity by 3000x under isoFLOP conditions, significantly improving long-context learning and retrieval efficiency.

2026-07-09 ~ 2026-07-10 · 4 related posts

• Sparse Memory Scales Linear RNN States — facebook · 2026-07-09
• Sparse Delta Memory Boosts RNN Long-Context Performance — TimDarcet · 2026-07-10
• Meta Releases Sparse Memory Module — teortaxesTex · 2026-07-10
• Sparse Delta Memory Boosts Linear RNN Memory — pmttyji · 2026-07-10