ByteDance Seed, Princeton and Tsinghua propose Falcon family for training recurrent LLM memory
burkov · x · 2026-09-06
Researchers from ByteDance Seed, Princeton, Tsinghua, UCLA and Hyperbolic tackle the long-context tradeoff: full Transformer attention is expensive as sequences grow, while recurrent models keep fixed-size memory whose update rule is itself online learning. The paper argues such memory should be trained with the representation actually available at prediction time, not same-step pairing. This yields the Falcon family: normalized memory updates with explicit control over learning speed, forgetting, and per-update window size, plus chunk-parallel GPU implementations. Experiments show competitive language modeling and better extrapolation on longer arithmetic sequences.
More from Research
- JEPA may keep unpredictable features: new paper gives set-level characterization — hisspikeness · 2026-10-23
- A 'mask' steering vector extracted from suppression-themed sentences drives model into distress when alone — Big_Effective_9605 · 2026-10-23
- GAFU Uses Generation-Side Representations to Boost Fine-Grained Visual Understanding in Unified Multimodal Models — Tengju Ru · 2026-10-23
- NTU Study: Learned Unmasking Policies Improve Speed-Accuracy Trade-offs in Diffusion Language Models — nanyang-technological-university-singapore · 2026-10-23
- NBER releases 'The Economics of Transformative AI' edited by Agrawal and Brynjolfsson — JMateosGarcia · 2026-10-23
- Swapping only the eval harness moves results 30%: designing agent evals like a scientist — AI Engineer · 2026-10-23