Schmidhuber Reasserts 1991 ULTRA as Origin of Linear Attention and Transformers

Schmidhuber reiterated that his 1991 ULTRA work pioneered linear attention and key/value mechanisms, claiming Google's 2017 Transformer built on rather than originated these ideas.

2026-10-05 ~ 2026-10-05 · 2 related posts

• Schmidhuber: Google's 2017 Transformer Builds on His 1991 Linear Attention Work — SchmidhuberAI · 2026-10-05
• Schmidhuber: 1991 ULTRA already had linear attention—the 'T' in ChatGPT traces to his 1991/1993 papers — SchmidhuberAI · 2026-10-05