FRAC replaces exponential forgetting with power-law memory in state space models
Ivan Kobyzev · hf · 2026-10-01
Ivan Kobyzev covers the FRAC paper: a new SSM architecture targeting the long-context memory bottleneck.
Problem
- SSMs compress sequence history into a bounded recurrent state; the memory decay law caps long-context performance. Most modern SSMs use ODE-based dynamics causing exponential forgetting.
Method
- FRAC, derived from fractional dynamics, replaces exponential decay with power-law long memory.
- It approximates the heavy-tailed kernel with a finite-state, log-spaced sum of exponential modes — an efficient recurrent module with parallel training and prefill, plus bounded-state autoregressive decoding.
Results
- Experiments including 1.3B-parameter language modeling show FRAC consistently improves long-context performance over SOTA SSM baselines while staying competitive at short context.
More from Research
- AI Overviews cites 38% fewer self-promotional listicles, drops brands 83% of the time: data — lilyraynyc · 2026-10-01
- 98,877 historical newspaper pages (1700s-1940s) with OCR text and word boxes hit Hugging Face — vanstriendaniel · 2026-10-01
- Microsoft's new ML system forecasts power-grid damage from space weather 30-60 min ahead — LesterMackey · 2026-10-01
- Noam Brown on Dwarkesh Podcast: Agent Swarms, Alignment, and Recursive Self-Improvement — GregKamradt · 2026-10-01
- Keras 3's quiet ZeroModels project packs 100+ model families across JAX, PyTorch, TensorFlow — lmoroney · 2026-10-01
- Initial probing of Opus suggests its time estimates are hallucinated from human training samples — StefanoGogioso · 2026-10-01