JHU composes four continual learning mechanisms, lifting long-horizon retention from 1.2% to 34.9%
DanielKhashabi · x · 2026-09-09
Researchers at Johns Hopkins (Zheyuan Zhang, Alvin Zhang; advised by Daniel Khashabi and Tianmin Shu) introduce ComposeCL, a simple recipe against catastrophic forgetting in continual learning:
- Core finding: composing multiple continual learning mechanisms substantially extends long-horizon memorization. Across 3 datasets with 100 sequential tasks each, average final retention rises from 1.2% to 34.9% vs naive sequential SFT — a 28x improvement.
- Design: a full 2⁴ factorial over four mechanisms — weight anchor (Synaptic Intelligence), function anchor (self-distillation), data anchor (generative replay), and merged LoRA — 16 combinations, 3 seeds each.
- Key results: replay and merged LoRA give the largest average gains, with joint benefit exceeding the sum of individual gains; the full composition ranks top-3 on all three datasets.
Paper, code, and datasets are open-sourced.
More from Research
- Researchers identify "Forking": train loss falls while validation loss turns up — ZimingLiu11 · 2026-10-26
- Google DeepMind pilots world's first double-blind evaluations for frontier models — scychan_brains · 2026-10-26
- Rewriting scaling law study: optimal synthetic data share rises from 10% to 32% with compute — heghbalz · 2026-10-26
- FLASH: GPU-native deformable sim learns cloth folding in minutes on one RTX 5090 — ChongZzZhang · 2026-10-26
- LEMON: zero-shot LLM misbehavior monitor flags hallucinations from attention maps — yoavartzi · 2026-10-26
- Behind the EMNLP keynote: LMs as simulated learners to generate child language acquisition hypotheses — najoungkim · 2026-10-26