Depth Memory framework unifies mHC and AttnRes, equipping layers with matrix-valued memory
orvieto_antonio · x · 2026-10-15
- Building on mHC and AttnRes showing that richer depth mixing improves performance at scale, the authors propose a unifying Depth Memory perspective: each layer reads from a set of streams to form its input, then writes its output back.
- Within this framework they introduce Linear Depth Memory (LDM), equipping models with a matrix-valued memory to explain why richer depth mixing helps.
Related event: Depth Memory Framework Unifies Benefits of Deeper Layer Mixing(2 posts)→
More from Research
- Curated roundup: frontier models help crack decades-old problems across disciplines — IgorCarron · 2026-10-15
- Stanford's CausalSmith automates causal inference research with Lean-verified proofs — StanfordAILab · 2026-10-15
- Cellular Intelligence wins Best Poster at scverse 2026 for perturbation platform — arjunrajlab · 2026-10-15
- BARRACUDA microfluidic chip traps a million single cells to map gene circuit design space at scale — NikoMcCarty · 2026-10-15
- Stanford's DexAgent turns one human egocentric video into dexterous robot training data — StanfordAILab · 2026-10-15
- Johns Hopkins wins HCOMP 2026 Best Paper for reader-centered summarization evaluation — mdredze · 2026-10-15