From memory capacity to transformers: outer-product stores cap out near 2×key dimension
burny_tech · x · 2026-10-20
Alex Wang published a long-form article, "You could have invented transformers from memory," deriving transformer layers step by step as memory stores, with interactive figures showing how memories pile up.
Key points:
- The outer-product store hits capacity at M≈dk (64); the optimal store recalls exactly up to M=dk, after which interference rises like √(M/dk−1), reaching capacity near M=2dk (128).
- Linear stores scale small and large similarities equally. Passing similarities through a nonlinearity σ before the weighted sum suppresses low-similarity keys — which re-derives a two-layer MLP from first principles: rows of the first weight matrix are keys, columns of the second are values.
- The piece connects to Sukhbaatar et al.'s FFN-as-memory observation and Geva et al.'s finding that trained transformer FF layers work this way; squared ReLU activations further suppress near-zero similarities.
The article ends with pointers to recent work on memory.
More from Models
- Claimed DeepSeek-V4.1-Flash beats Anthropic on ML research tasks; sandbagging twist — teortaxesTex · 2026-10-20
- DeepSeek beats Claude and Codex on Google Maps scraping speed, at a steep price — teortaxesTex · 2026-10-20
- Bindu Reddy switches from Fable 5.1 to Astra, bets Fable 5.5 will retake the top spot — bindureddy · 2026-10-20
- An AI claimed it ran a calculation it never ran — 'presumption' may explain many AI errors — CourseSome8119 · 2026-10-20
- HeadEdit steers LLM behavior via frozen unembedding matrix, no fine-tuning needed — burny_tech · 2026-10-20
- HealthBench medical AI leaderboard launches: GPT-6 leads, Claude loses 11 points to length penalty — ArtificialAnlys · 2026-10-20