A GD-trained linear layer is exactly linear attention over its own training history
ShahabBakht · x · 2026-10-12
mtavitschlegel highlights a beautiful but little-known dual view: a linear layer trained with gradient descent computes exactly the same function as its initial weights plus linear attention over its own training history — each test input attends to every training input (as keys) and recombines the backpropagated errors stored with them (as values).
The connection has been discovered and revisited before but remains poorly known. Taking it seriously and replacing the linear attention with more powerful kernels (e.g., softmax attention — nontrivial in practice) leads to a new paradigm. The attached slide, from Google's "Paradigms of Intelligence" talk on Retrieval centric deep learning in growing nonparametric networks, lays out the linearization-to-nonlinearization derivation and the shift from W·x to attention(K, V, x).
Related event: Gradient-Trained Linear Layers Are Equivalent to Linear Attention(2 posts)→
More from Models
- Ant Group Releases PolyOCR, Unified OCR Foundation Models with SOTA Results — GuangJian Team · 2026-10-12
- Leak claims OpenAI's next models: 5T-param sol on spud, 10T-param astra on doug pre-train — GregCook2011 · 2026-10-12
- Dev vents: OpenAI models still suck at writing, why is that? — beechinour · 2026-10-12
- Musk on Grok RL traces: 'like a person undergoing torture,' vows more care — Singularitarian · 2026-10-12
- OCR evals done right: custom annotation apps and agent citation checkers to catch silent errors — HamelHusain · 2026-10-12
- NYU Mathematician Tristan Buckmaster: OpenAI Made Zero Progress on Navier-Stokes Until They 'Learned His Name' — AlexTensor · 2026-10-12