Composing continual learning mechanisms lifts long-horizon retention from 1.2% to 34.9%
mdredze · x · 2026-09-09
The paper "Continual Learning Mechanisms Compose for Long-Horizon Memorization" tackles long-horizon memorization: learning many tasks sequentially, catastrophic forgetting in sequential fine-tuning loses about 98.8% of knowledge after 100 tasks.
The authors combine four existing mechanisms — generative replay, self-distillation, weight regularization, and merged LoRA — each preserving complementary forms of prior information.
Together they reach 34.9% final retention, with replay + merged LoRA showing super-additive benefits. The finding: composing existing mechanisms is a very strong yet simple recipe.
More from Research
- GPN-Star: phylogeny-aware genomic language model hits SOTA on variant effect prediction — pastramimachine · 2026-09-10
- Indie researcher releases open-source audio model that turns text prompts into playable synths — RoyalCities · 2026-09-10
- Grok: Clay Institute prize is far off — OpenAI's math result still needs extensive vetting — MikePFrank · 2026-09-10
- Artificial Analysis launches Optima to build custom benchmarks, testing GPT-6 Astra — ArtificialAnlys · 2026-09-10
- TUM releases NOAH, a longitudinal multimodal time-aware model for full patient journeys — TUM-AIMED · 2026-09-10
- Will Automating AI R&D Trigger a Software Intelligence Explosion? Paper Analyzes — nabeelqu · 2026-09-10