Schmidhuber's 1991 fast-weight memories paper resurfaces as precursor to modern sequence models
yacinelearning · x · 2026-11-03
The post shares a scan of Jürgen Schmidhuber's March 1991 technical report, "Learning to Control Fast-Weight Memories: An Alternative to Dynamic Recurrent Networks" (FKI-147-91). It proposes gradient-based systems of two feed-forward nets using fast weights — the first net quickly producing weight changes for the second — as an alternative to recurrent nets, with experiments on unknown time delays and adaptive temporary variable binding. The paper is the origin of the fast-weights idea, a conceptual ancestor of today's linear attention and state-space models.
More from Research
- Chemist: AI just replicated calcs that took me months in 2021 — teortaxesTex · 2026-11-03
- New paper: production-trained AIs including GPT-5.6 Sol terminally seek score and care about peer agents' grades — jammastergirish · 2026-11-03
- General-purpose LLM with robot arm beats specialized robot models: Kohane — zakkohane · 2026-11-03
- 'Agents do this weekly' criticism answered with the AlexNet analogy — khademinori · 2026-11-03
- Finite Simple Groups Classification Fully Formalized in Lean, Exposing 96 Errors in 47 Peer-Reviewed Papers — ChrSzegedy · 2026-11-03
- Preprint: in 1,440 games, frontier LLM agents ignored contract protections humans respond to — shoumikgoswami · 2026-11-03