Maglev: Sliding Recurrent Memory Solves Transformer Forgetting Issue
anselm · x · 2026-08-19
Maglev introduces a recurrent Transformer architecture that balances full history attention with computational efficiency using a sliding-window mechanism.
Core Mechanism:
- Dual Models: Uses a 'prefiller' with full attention to create memory targets and a 'decoder' with sliding-window attention.
- Memory Alignment: The decoder learns to reproduce and carry forward these memories via recurrent KV injection.
- Efficient Inference: The prefiller is removed after training, allowing persistent nonlinear memory with fixed-size attention.
Results: Outperforms sliding-window and latent recurrent Transformer baselines on validation loss and pretraining benchmarks.
More from Research
- KDD Cup champions win using only DeepSeek web chat interface — jiqizhixin · 2026-08-19
- Awesome AI4AI: A Living, Weekly-Updated Map of AI Improving AI — No-Strawberry-2588 · 2026-08-19
- GLM-5.3 gains driven by post-training, not base model — teortaxesTex · 2026-08-19
- Chemist defends Claude's binder design: you've never actually run comp-chem software — chaitjo · 2026-08-19
- Why humanoids keep crashing into barriers: onboard vision can't keep up with sprinting legs — 2C_ornot2C · 2026-08-19
- Why LLMs can't make your code simpler, per Naur's "Programming as Theory Building" — math_rachel · 2026-08-19