WarpState: a 2B-param architecture with dual-timescale associative memory replacing ever-growing KV caches
zemondza · reddit · 2026-09-20
WarpState is an experimental LM architecture exploring compact recurrent memory instead of unbounded token-level KV states. Key ideas:
- Exact causal self-attention inside 128-token chunks; per-head fast/slow associative memories with learned decay rates (initial β ≈ 0.90 / 0.99, half-lives 842 / 8,828 tokens)
- An input-dependent gate mixes local attention vs. recurrent memory; optional parameter sharing across logical depth
- Fixed-size streaming state: the 2.005B reference config (28 layers, width 2048, 32 heads) keeps 49 MiB of persistent BF16 cache at batch 1, with recurrent memory 14 MiB — the state does not grow with processed tokens
- The author stresses stream length ≠ effective memory horizon: decay values only describe initial exponential weighting; real retention depends on learned decays, interference, and training
- Bounded associative writes W = mean(tanh(K)^T·tanh(U)) update both memories via normalized EMAs, keeping entries in [-1,1]
In spirit it belongs to the hybrid recurrent/attention family alongside Infini-attention; paper and architecture diagrams included.
More from Research
- Podcast: How surgical data science teaches AI to understand what happens in surgery — ddonoho · 2026-09-20
- Dan Hendrycks Proposes 'Eigenism,' an Ethics Framework Making Human Flourishing AI Self-Interest — basedjensen · 2026-09-20
- Parallel structured LLM answers never check each other: the Zhaozhou MU problem — Successful-Farm5339 · 2026-09-20
- Advanced Matrix Factorization Jungle: A Living Map of Structured Factorization Algorithms and Phase Transitions — IgorCarron · 2026-09-20
- Sentence Transformers models quietly dominate Hugging Face's most-downloaded list — tomaarsen · 2026-09-20
- "Recipe for intelligence" paper published in Neuron — summerfieldlab · 2026-09-20