ByteDance Seed, Princeton and Tsinghua propose Falcon family for training recurrent LLM memory
burkov · x · 2026-09-06
Researchers from ByteDance Seed, Princeton, Tsinghua, UCLA and Hyperbolic tackle the long-context tradeoff: full Transformer attention is expensive as sequences grow, while recurrent models keep fixed-size memory whose update rule is itself online learning. The paper argues such memory should be trained with the representation actually available at prediction time, not same-step pairing. This yields the Falcon family: normalized memory updates with explicit control over learning speed, forgetting, and per-update window size, plus chunk-parallel GPU implementations. Experiments show competitive language modeling and better extrapolation on longer arithmetic sequences.
More from Research
- Boltz Lab reveals 50x molecular token growth in 5 months as Series A closes — GabriCorso · 2026-11-19
- Researcher calls METR/Anthropic '2x speedup' RSI analysis statistically dubious — jessi_cata · 2026-11-19
- Linear Probes Could Reveal Which Persona a Base Model Is Playing, Researchers Suggest — cephaloform · 2026-11-19
- ETH's Anymal robot completes 10km autonomous alpine hike with VLM-guided ROAM system — ChongZzZhang · 2026-11-19
- OpenVax codebase grows from 78K to 565K lines in 15 months — iskander · 2026-11-19
- François Fleuret: decoder transformer's elegance — no trade-off hyperparameters — francoisfleuret · 2026-11-19