RNN memory is set by recurrent matrix eigenvalues — LRU's ring init fixes it
RobertTLange · x · 2026-11-15
Robert TLange explains the math behind what an RNN 'remembers':
- Eigenvalues |λ| near 0 → forgets within a few steps
- |λ| near 1 → long memory
- |λ| > 1 → activations explode (chaotic dynamics for tanh)
The Linear Recurrent Unit (LRU) fixes this by initialising all eigenvalues on a thin ring just inside the unit circle (e.g. |λ| ∈ [0.9, 0.999]), giving long memory that is stable by construction. A toy simulation with 100 units and a random Gaussian matrix illustrates the dynamics.
References cited: Sompolinsky et al. (1988) on chaos in random neural networks (Phys. Rev. Lett.), and Orvieto et al.'s 2023 LRU paper.
Related event: RNN Memory Explained: Recurrent Matrix Eigenvalues and LRU Initialization(2 posts)→
More from Models
- After a week with Carbon: smarter than Opus 5.5, but the best workflow pairs both — rudrank · 2026-11-15
- User builds entire single-file project with Gemini — no libraries, no assets — fofrAI · 2026-11-15
- GPT-6.2 rumor timeline: the rumor was late, not the model — lxfater · 2026-11-15
- OpenAI falling-behind narrative 'massively exaggerated', claims insider: Astra still competitive with Opus 5.5 — haider1 · 2026-11-15
- EU's Open Model EUROPA Eyes ~1T Params, Per-GPU Throughput Up 14x — mawcopolow · 2026-11-15
- Claude mobile web bug: text inside antThinking tags silently stripped after sending — Kyrannio · 2026-11-15