Schmidhuber: the 1991 Fast Weight Programmers were the first Transformer variants
SchmidhuberAI · x · 2026-11-03
Schmidhuber reiterates his long-standing claim that Transformers are nothing new: his 1991 Fast Weight Programmers work included what is now called the unnormalised linear Transformer, mathematically equivalent to modern linear Transformers, with linear rather than quadratic cost scaling. The same year he introduced self-supervised pre-training for deep NNs (the 'P' in GPT). The post links to his blog overview with the original papers — another installment in his historical priority claim on Transformer architecture.
More from Research
- Chemist: AI just replicated calcs that took me months in 2021 — teortaxesTex · 2026-11-03
- New paper: production-trained AIs including GPT-5.6 Sol terminally seek score and care about peer agents' grades — jammastergirish · 2026-11-03
- General-purpose LLM with robot arm beats specialized robot models: Kohane — zakkohane · 2026-11-03
- 'Agents do this weekly' criticism answered with the AlexNet analogy — khademinori · 2026-11-03
- Finite Simple Groups Classification Fully Formalized in Lean, Exposing 96 Errors in 47 Peer-Reviewed Papers — ChrSzegedy · 2026-11-03
- Preprint: in 1,440 games, frontier LLM agents ignored contract protections humans respond to — shoumikgoswami · 2026-11-03