Raven: Linear-Time Sequence Model Achieves High-Recall via Sparse Memory Routing
bronzeagepapi · x · 2026-08-01
A new paper titled Raven: High-Recall Sequence Modeling with Sparse Memory Routing by Albert Gu and colleagues introduces Raven, a novel linear-time sequence model.
The model addresses the tradeoff in long-context recall: state-based linear models like SSMs write densely causing interference, while Sliding Window Attention (SWA) evicts old tokens entirely due to fixed windows. Raven maintains a fixed set of memory slots and, at each step, decays and updates only a selected subset via learned, input-dependent routing.
This approach mitigates SWA's hard eviction and SSMs' dense interference. Across recall-intensive benchmarks, Raven outperforms prior baselines and remains effective when extrapolating to context lengths up to 16x its training length.
More from Research
- Research Presentation: Empathic Communication and Augmenting Human Potential with AI — mattgroh · 2026-08-01
- GymAnything: Automatically Building 12,000+ RL Environments for Computer-Use Agents — pratyushmaini · 2026-08-01
- Laplace Transform Lecture Explains the Math Behind Why Mamba Works — JosephJacks_ · 2026-08-01
- Hugging Face Release: Open Pangram Models & Datasets Based on ICLR Paper — AaronBergman18 · 2026-08-01
- Emulated: Training Fully Autonomous Software Engineers in Full-Company Simulations — AI Engineer · 2026-08-01
- List of NeurIPS 2026 Workshops on Health, Biology, and Scientific Discovery — DanielKhashabi · 2026-08-01