Memory Attention: Paper Replaces Value Projection With Lookup, Cuts GPU Storage 7.38%
serrjoa · x · 2026-09-27
A new alphaXiv paper by Jiale Kang proposes Memory Attention (MA), replacing the attention value projection with a token-indexed lookup.
Core idea
- Instead of computing fresh values per token, MA retrieves a layer-specific memory M by token ID and forms V = K + M: the memory supplies reusable token representations while contextual keys preserve context dependence
- At inference, normalization folds into memory tables, reducing value construction to lookup plus addition
Engineering highlights
- MA-Offload stores memory on CPU with prefetching: the prototype carries 2.08× total parameters while cutting GPU parameter storage by 7.38% at near-baseline latency; SSD offloading is next
- MA-Recall direction: keep only K and token IDs, reconstruct V on demand — potentially eliminating the persistent V cache
Under matched training token budgets, experiments across attention configurations show improved language modeling and downstream performance. The author, inspired by Engram, frames this as a candidate direction for future model architectures.
More from Infra
- India's Sarvam launches Token Factory, hosting top open models on local infra — HarveenChadha · 2026-11-16
- Gemini 3.5 Flash now runs on air-gapped Google Distributed Cloud, offline up to 12 months — shashib · 2026-11-16
- Gemma.c: full Gemma 1/2/3 multimodal inference in a single C file, ~72 tok/s on CPU at INT8 — This-Purchase-3325 · 2026-11-16
- BIS Issues Interim Final Rule Addressing Remote Access to Advanced AI Chips — apples_jimmy · 2026-11-16
- Jailbroken PS4 runs a 9B open-source model at up to 43 tok/s by talking directly to its 2013-era GPU — AIFlow_ML · 2026-11-16
- Mobile AI Factory: agents auto-build on-device SLMs with 19x less GPU time — pmttyji · 2026-11-16