Alex Wang's Perplexity reading group slides reframe Transformer design through associative memory
andrewgwils · x · 2026-11-06
Alex Wang presented "Associative Memory" at Perplexity's reading group (invited by Andrew Wilson), publishing full slides that offer a unifying lens on Transformer architecture choices.
Key points:
- Components like ReLU/GELU/SwiGLU, short convolutions on keys/values, attention sinks, linear attention, DeltaNet, and SSMs are usually described by what they compute, rarely by what they're for — making design choices feel arbitrary.
- He argues associative memory, an old idea from Kohonen and Hopfield (who shared the 2024 Nobel Prize in Physics), provides a language for what each part is trying to do.
- Starting from outer-product memory stores (W = Σ vᵢkᵢᵀ) with recall f(x) = Wx, orthonormal keys give clean recall while non-orthogonality introduces interference — explaining the motivation behind many modern architecture tweaks.
- Builds on his blog "You could have invented transformers from memory" and the JMLR paper "Test-time regression", connecting to fast weights, test-time training, SSMs, and GDN.
More from Research
- NextGAN paper hits FID 1.36 one-step on ImageNet 256, scales to 14B video models — serrjoa · 2026-11-06
- SLA scales spatial-language-action models via continual pre-training with frozen 3D alignment — HKUSTGZ · 2026-11-06
- Harness-S rescues failed RL rollouts with sibling harnesses for generalizable agents — SJUT1 · 2026-11-06
- VisSearch-Bench: best multimodal agents fully solve only 26.47% of visual search tasks — antgroup · 2026-11-06
- Identically trained LLMs break on different jailbreaks: pretraining seeds dominate — cerulean-works · 2026-11-06
- KAIST's DreamFree distills video diffusion representations for real-time robot control — kaist-ai · 2026-11-06