Domingos: An attention head is just a fast-weight MLP
pmddomingos · x · 2026-10-24
Pedro Domingos distills a recent paper into one line: an attention head is a fast-weight MLP. The linked section 3.1 proves scaled dot-product attention is exactly equivalent to a two-layer, N-width MLP whose weights are the scaled keys/values and whose activation is exp(·), recasting efficient attention methods as different ways of scaling fast weights.
More from Research
- Methodologist argues placebo observables are a bad sign for IQ as a measurement — RexDouglass · 2026-10-24
- 'LLMs as Human Proxies' paper lands Nature Computational Science cover — iyadrahwan · 2026-10-24
- DOE's Genesis Mission Lists 297 Projects, to Release Open-Weight Foundation Models — ingliguori · 2026-10-24
- Kornia 0.9.0 ships a conventions page targeting LLM coding pitfalls — kornia_foss · 2026-10-24
- Fleuret: teacher forcing turns time iteration into depth iteration, and GPUs love it — francoisfleuret · 2026-10-24
- Ross Taylor: studying LLM behavior will increasingly resemble social science — rosstaylor90 · 2026-10-24