Gradient-Trained Linear Layers Are Equivalent to Linear Attention

A new paper in collaboration with Google shows that a linear layer trained by gradient descent is exactly equivalent to its initial weights plus linear attention over its own training history, leading to a novel retrieval-centric learning paradigm derived from the SGD–linear attention duality.

2026-10-11 ~ 2026-10-12 · 2 related posts

• New Paper Derives a Learning Paradigm from the SGD–Linear Attention Duality — HanGuo97 · 2026-10-11
• A GD-trained linear layer is exactly linear attention over its own training history — ShahabBakht · 2026-10-12