Linear Depth Memory: matrix-valued memory tops PreLN and mHC on DepthBench across scales
SonglinYang4 · x · 2026-10-15
- A new Depth Memory framework offers a unifying perspective: each layer reads from a set of streams to form its input, then writes its output back. Prior work on mHC and AttnRes showed richer depth mixing improves performance at scale; this framework aims to explain why.
- Linear Depth Memory (LDM) equips models with a matrix-valued memory. Perplexity charts at 400M and 1.6B scales across width/depth configurations show LDM beating PreLN, AttnRes (Block/Full), HC, and mHC, with the gap persisting as scale grows.
Related event: Depth Memory Framework Unifies Benefits of Deeper Layer Mixing(2 posts)→
More from Research
- RISE (NeurIPS 2026): evolving attack strategies to break through text-to-image guardrails — rohanpaul_ai · 2026-10-15
- Random Brainfuck programs on a 2D grid spontaneously evolve self-replicators — spikedoanz · 2026-10-15
- Classifier run on 42,000+ ICLR 2027 submissions finds over half nearly fully AI-assisted — nsaphra · 2026-10-15
- LLM4Code workshop at ICSE 2027 opens call for papers, due Nov 13, 2026 — LingmingZhang · 2026-10-15
- ELSA3D, a unified 3D generation and captioning model, hits SOTA at half the FLOPs, accepted at NeurIPS — CSProfKGD · 2026-10-15
- Moving ViT weights 1.3% without changing predictions scrambles representations, challenging Platonic hypothesis — neuroecology · 2026-10-15