Maglev: Sliding Recurrent Memory for Parallel Training and Sequential Decoding
ChengleiSi · x · 2026-08-12
Maglev is a recurrent Transformer architecture with fixed-size memory designed to enable parallelizable training alongside sequential decoding.
- Architecture: It consists of a prefiller $Q$ utilizing full attention to produce memory targets, and a decoder $P$ using only sliding-window attention with recurrent K/V injection for next-token prediction.
- Training & Inference: Trained with a memory consistency loss aligning $P$'s states with $Q$'s targets. During inference, the teacher $Q$ is dropped entirely, and the student $P$ rolls its memory forward sequentially, closing the train-inference gap.
- Results: Empirically improves validation loss and downstream pretraining benchmarks over sliding-window and latent recurrent baselines. Sharing parameters between $P$ and $Q$ also reduces memory footprint while preserving most gains.
More from Research
- Study Finds: Higher Reasoning Effort Makes LLMs Less Likely to Ask Clarifying Questions — xuanalogue · 2026-08-12
- Open Dataset Measures AI's Actual Impact on Accelerating Scientific Discovery — soumitrashukla9 · 2026-08-12
- Six Months of AI Auto-Research Tools, But No Clear Acceleration in Algorithmic Efficiency — soumitrashukla9 · 2026-08-12
- Research Reveals the Personality Evolution of the Grok Model Family — DevDminGod · 2026-08-12
- STACX: A Modular Infrastructure for End-to-End Agentic RL — daibond_alpha · 2026-08-12
- NeurIPS 2026 Workshop: World Models for High-Stakes Healthcare — yaringal · 2026-08-12