Memory Attention: Paper Replaces Value Projection With Lookup, Cuts GPU Storage 7.38%
serrjoa · x · 2026-09-27
A new alphaXiv paper by Jiale Kang proposes Memory Attention (MA), replacing the attention value projection with a token-indexed lookup.
Core idea
- Instead of computing fresh values per token, MA retrieves a layer-specific memory M by token ID and forms V = K + M: the memory supplies reusable token representations while contextual keys preserve context dependence
- At inference, normalization folds into memory tables, reducing value construction to lookup plus addition
Engineering highlights
- MA-Offload stores memory on CPU with prefetching: the prototype carries 2.08× total parameters while cutting GPU parameter storage by 7.38% at near-baseline latency; SSD offloading is next
- MA-Recall direction: keep only K and token IDs, reconstruct V on demand — potentially eliminating the persistent V cache
Under matched training token budgets, experiments across attention configurations show improved language modeling and downstream performance. The author, inspired by Engram, frames this as a candidate direction for future model architectures.
More from Infra
- DRAM output ~50EB vs NAND ~1100EB this year; CXL pooling is about utilization, not replacing eSSD — zephyr_z9 · 2026-09-27
- Bought a 64GB DDR5 kit 3 years ago for 17k rupees — now it's worth 90k as RAM prices soar — ojasvi_yadav · 2026-09-27
- Thermocompression bonding a 200mm wafer: 70 kN at 400-450°C for 20-45 min — jwt0625 · 2026-09-27
- OpenAI reportedly facing ugly compute shortage as Pro quotas become the new Plus — ns123abc · 2026-09-27
- UK village revolts against 1.5GW AI datacentre planned in UNESCO biosphere reserve — nordicinst · 2026-09-27
- Bittensor GPU rental project bills $964k in a month, growing 36% MoM — markjeffrey · 2026-09-27