Schmidhuber reasserts his 1991 linear Transformer predates and underlies Google's 2017 architecture
SchmidhuberAI · x · 2026-10-23
Jürgen Schmidhuber reasserted his long-running priority claim: Google's 2017 normalized quadratic Transformer (TR1) is based on principles from his 1991 unnormalized linear Transformer (ULTRA), where KEY/VALUE was called FROM/TO, and computational costs scale linearly rather than quadratically in input size.
He also notes his 1993 recurrent extension paper (FWP2) introduced the attention terminology — learning "internal spotlights of attention" by gradient descent. It's one of AI history's most famous priority disputes, and he regularly argues the T in ChatGPT has deeper roots.
More from Research
- OpenResearch integrates Trackio: agents log experiments and read curves to plan next runs — Gradio · 2026-10-24
- Owain Evans: Turning intuitions into concepts is hard — AI self-introspection will be too — OwainEvans_UK · 2026-10-24
- Alien tops human-judged research report quality, scales to 25,000 AI-generated research proposals — anirudhg9119 · 2026-10-24
- Nanograb's AI-designed nanoparticles target mRNA delivery by cell receptor fingerprint — ycombinator · 2026-10-24
- Anthropic Blog Revives lambda py, a 2013 Brown MOOC Spinoff, in Python Interpreter Case Study — tianyin_xu · 2026-10-24
- LitPivot surfaces overlaps and gaps in prior work, suggests revisions — allen_ai · 2026-10-24