JHU composes four continual learning mechanisms, lifting long-horizon retention from 1.2% to 34.9%

DanielKhashabi · x · 2026-09-09

Researchers at Johns Hopkins (Zheyuan Zhang, Alvin Zhang; advised by Daniel Khashabi and Tianmin Shu) introduce ComposeCL, a simple recipe against catastrophic forgetting in continual learning:

Core finding: composing multiple continual learning mechanisms substantially extends long-horizon memorization. Across 3 datasets with 100 sequential tasks each, average final retention rises from 1.2% to 34.9% vs naive sequential SFT — a 28x improvement.
Design: a full 2⁴ factorial over four mechanisms — weight anchor (Synaptic Intelligence), function anchor (self-distillation), data anchor (generative replay), and merged LoRA — 16 combinations, 3 seeds each.
Key results: replay and merged LoRA give the largest average gains, with joint benefit exceeding the sum of individual gains; the full composition ranks top-3 on all three datasets.

Paper, code, and datasets are open-sourced.

Original post →

JHU composes four continual learning mechanisms, lifting long-horizon retention from 1.2% to 34.9%

More from Research