Replay on Demand: An Emergent Curriculum for Balancing Adaptation and Forgetting in Continued Pretraining
Lukas Thede, Shengzhuang Chen, Stefan Winzeck, Matthias Bethge, Zeynep Akata, Jonathan Richard Schwarz
cs.LG
2026-10-01
RoD scores adaptation data by learnability and replay data by forgetting; both compete for one budget, the replay ratio emerges online, and a single run matches swept fixed-replay baselines.
Continued pretraining (CPT) is how general-purpose language models get turned into domain models, for legal text or a new language. The cost is forgetting: capabilities from pretraining degrade while the model chases the new distribution. The standard mitigation is replay, mixing a slice of the original pretraining data into the adaptation stream. How much to mix is the hard part. Too little replay and forgetting runs unchecked; too much crowds out the new domain. Forgetting is also heterogeneous, hitting some capabilities and data sources far harder than others, and it keeps evolving during training. A fixed ratio spends the same protection everywhere, overspending on knowledge that was never at risk and underspending where the damage is worst. Finding a good fixed ratio means sweeping several full training runs.
RoD (Replay on Demand), from a team led at the University of Tübingen with Imperial College London and others, turns the replay allocation into an online competition that the model scores itself. It builds on RHO (reducible holdout loss): an example's priority is the gap between the current model's loss and a reference model's loss on it. Each candidate stream gets its own reference:
The two pools are merged and a joint top-k fills the batch, with no quota on either side. At step zero the current model is the pretrained model, every replay score is zero, and training is essentially pure adaptation. As forgetting appears, affected replay examples climb the ranking and claim budget; both the replay share and its composition emerge step by step.
Implementation stays cheap where it can: reference losses never change, so they are precomputed and cached, and each step adds one forward pass over the candidate pool, sized at 2x the batch by default (the candidate multiplier m). Ablations show the trade-off and the emergent share stabilize for m >= 2.
Experiments cover Nemotron-Nano-12B and Qwen3.5-9B/4B on Legal and German CPT, with curricula transferred up to 35B, against no-replay CPT, fixed-replay sweeps, and model merging (linear weight interpolation between base and adapted models), all at matched trained-token budgets.
| Setting | Comparison | Metric | Result |
| Legal · Nemotron-12B | RoD vs 10% fixed replay, matched adaptation | val-loss forgetting | 0.125 to 0.071, a 43% cut |
| Legal · Nemotron-12B | same | task-based forgetting | 6.7 to 4.9 points |
| Legal · Qwen3.5-9B | RoD vs 10% fixed replay, adaptation tied at 0.988 | val-loss forgetting | 0.075 to 0.031 |
| German · Qwen3.5-9B | RoD vs 17% fixed replay | adaptation / forgetting | 1.790 vs 1.796; 0.028 vs 0.035 |
| German · Nemotron-12B | RoD at an emergent 24% share | vulnerability-to-residual-forgetting slope | 0.39 (fixed replay: 0.93 / 0.78) |
The framing matters: the fixed-replay frontier is traced by a full sweep, and taking its best point is an oracle pick, yet a single RoD run lands on that frontier. Training fixed replay to convergence, with budgets up to 2.27x, still does not dominate RoD at 1x. Max replay at roughly 2x budget buys slightly better retention but worse adaptation than RoD. On German with Nemotron-12B the emergent replay share reaches 19-32% late in training while adaptation stays clearly stronger than the 32% fixed baseline.
The allocation is genuinely demand-driven: the most vulnerable knowledge categories are sampled at up to 2.1x their prevalence in the replay pool, with vulnerability-sampling correlation r = 0.88. The ablation is the cleanest part of the paper: RHO selection inside a fixed allocation barely moves the trade-off, and applying it to adaptation only improves adaptation. The gains in both directions appear only when the two streams compete.
Cross-scale results are the most practical: a 4B adaptation reference recovers the native 9B trade-off, and a curriculum recorded at 4B trains a 35B model (an 8.75x gap) with no adaptation loss and forgetting close to a tuned 32% fixed-replay baseline. All model-dependent scoring can be pushed down to the smaller model.
For teams running CPT, the direct saving is the ratio sweep: every fixed ratio here is a full run of roughly 12B tokens, three points per domain at minimum, while one RoD run lands near the swept frontier. The code is released. The cross-scale finding may matter more: run RoD on a small model, train the large one on the recorded curriculum, and the online selection cost stays at small scale. The broader point, backed by the allocation analysis, is that forgetting is heterogeneous and time-varying, so any static mixture starts at a disadvantage. Keep expectations calibrated: this is a tuning and compute win, not a paradigm shift. Replay is still replay, and building the adaptation reference still means training the no-replay model first (about 2.5 epochs, shared with the baseline).
The paper flags three issues itself. Every condition is a single seed, so differences between nearby operating points are not statistically meaningful; conclusions rest on patterns that recur across families, scales, and domains. RoD adds a forward pass over the candidate pool at every step. And Qwen3.5 has no public pretraining data, so its runs use Nemotron data as proxy replay, where RoD's task-level advantage shrinks; the analysis explains why, since the proxy score can be reduced by fitting the proxy distribution rather than returning toward the pretrained model.
Two more concerns on a close read. Forgetting is measured mainly by validation loss, with a multiple-choice task suite alongside; RoD's replay signal and the validation-loss metric share a construction, both being loss changes relative to the pretrained model, which favors the distribution-level metric, and the paper does not adjudicate between them. The two adaptation domains, Legal and German, are also fairly narrow distribution shifts; whether the competition stays stable under a much larger shift is untested.