LIMe cuts LLM training compute by 15% with layer-integrated attention memory
ChampionshipLimp1749 · reddit · 2026-10-29
Researchers from T-Tech presented LIMe (Layer-Integrated Memory) at EMNLP 2026. Traditional Transformers force all context through the previous layer's hidden state, causing representation collapse. LIMe adds learnable per-head routing weights that let attention heads retrieve key-value buffers from earlier layers, reusing computed information with unchanged hidden-state size. On 1B-parameter models it matched LLaMA's training loss with 15% less compute, beat competing methods on language modeling and multi-step reasoning, and showed highest gains on arithmetic tasks and deeper architectures. Paper: arxiv.org/abs/2502.09245; open-source implementation available.
More from Research
- Transformer layers turned into a full-fledged symbolic CPU — yshan2u · 2026-10-29
- NanoDreamer: A 750K-parameter world-model agent trained on a MacBook M5 Pro plays 5 MinAtar games — CatAstro_Piyush · 2026-10-29
- Why You Can't Just Remove Positional Embeddings: Self-Attention Is Permutation Equivariant — cneuralnetwork · 2026-10-29
- MIT develops injectable gel that may repair damaged nerves, restoring sensation — CurieuxExplorer · 2026-10-29
- OmniSpectra: a wavelength-native spectral foundation model that skips destructive resampling — Md Khairul Islam · 2026-10-29
- Two curated sites to track Lean: Papers With Lean and Lean Weekly — TZahavy · 2026-10-29