Schmidhuber revives his 1991 fast weight programmers: the first linear Transformer variant
SchmidhuberAI · x · 2026-11-03
Jürgen Schmidhuber reposted his blog essay arguing that today's linear attention work is "old hat": his 1991 Fast Weight Programmers were the first Transformer variant — an unnormalised linear Transformer that is mathematically equivalent to modern approaches, with costs scaling linearly rather than quadratically due to compute constraints of the era. The same year he also introduced self-supervised pre-training for deep networks, the "P" in GPT. The post connects fast weights to metalearning, where a feedforward net learns to program the fast weights of another net, separating storage and control like a classical computer.
Related event: Schmidhuber Reiterates Fast Weights as 1991 Transformer Precursor(2 posts)→
More from AGI Musings
- Software is shifting from winner-take-all to a restaurant-style long-tail market — tinyfool · 2026-11-03
- The Russell Conjugations of Intelligence: Why 'Just Matrix Multiplication' Offends Us for Brains but Not for AI — esrtweet · 2026-11-03
- AI safety researcher: assuming all emergence is risk may be the wrong frame — repligate · 2026-11-03
- AI bubble burst favors pure ML backgrounds: end-to-end skills now beat niche expertise — kmeanskaran · 2026-11-03
- Red teamer: jailbreak work isn't ethically neutral and takes a toll on humans and models — repligate · 2026-11-03
- repligate: "AI welfare" was fringe a year ago, and its framing is already a concession — repligate · 2026-11-03