JHU's ComposeCL: Combining Continual Learning Mechanisms Boosts Long-Horizon Memory Retention

A Johns Hopkins team (Zheyuan Zhang, Alvin Zhang, et al., advised by Daniel Khashabi and Tianmin Shu) released ComposeCL, a paper titled "Continual Learning Mechanisms Compose for Long-Horizon Memorization" studying long-horizon memorization in language models: models must sequentially learn 100 QA tasks via continual fine-tuning, without keeping early training samples or task identities, while absorbing new facts and retaining old knowledge. Experiments show naive sequential fine-tuning suffers catastrophic forgetting, retaining only about 1.2% of old knowledge after 100 tasks; none of the single continual learning mechanisms evaluated adequately solves the problem. The key finding is that a simple combination of multiple continual learning mechanisms substantially improves retention to 34.9%. Andrey Burkov also relayed the work, noting that sequential supervised fine-tuning overwrites earlier associations — the core pain point in continual learning settings.

Confirmed

• The research comes from Johns Hopkins; paper title: "Continual Learning Mechanisms Compose for Long-Horizon Memorization"
• Setup: sequential fine-tuning over 100 QA tasks, no replay of old samples, no task identity
• Naive sequential fine-tuning retains 1.2% of knowledge after 100 tasks; single mechanisms are insufficient; combining mechanisms raises retention to 34.9%

Why it matters

• Long-horizon memorization is key to letting models absorb new knowledge without forgetting old knowledge, directly affecting whether models can be updated incrementally over time
• Results show existing single continual learning methods are insufficient at the hundred-task scale, while combining mechanisms is a simple and effective improvement path

2026-09-09 ~ 2026-09-10 · 5 related posts

• Composing Continual Learning Mechanisms Boosts 100-Task Memorization Retention 28x to 34.9% — DanielKhashabi · 2026-09-09
• Composing continual learning mechanisms lifts long-horizon retention from 1.2% to 34.9% — mdredze · 2026-09-09
• JHU composes four continual learning mechanisms, lifting long-horizon retention from 1.2% to 34.9% — DanielKhashabi · 2026-09-09
• Sequential fine-tuning leaves models with just 1.2% accuracy on old tasks after 100 updates — burkov · 2026-09-10

1 near-duplicate retellings: DanielKhashabi