CMU: Model Merging and 'Grafting' Let Post-Trained LLMs Learn Off-Policy Without Forgetting
AdtRaghunathan · x · 2026-11-03
CMU researchers (Chen Henry Wu, Thomas Zhang, Aditi Raghunathan) published a blog on continual learning for LLMs, based on their paper 'Off-Policy Merging Beats On-Policy Self-Distillation for Continual Learning' with open-source code. Post-trained (RL) models degrade when SFT'd on new data—even learning harder math can hurt easier math. Since new knowledge is inherently off-policy, converting it to on-policy signals (e.g., self-distillation) causes artifacts like suppressed uncertainty. They show simple model merging makes off-policy SFT surprisingly strong, and a new 'grafting' method—learning updates on earlier checkpoints and transferring them to post-trained models—largely removes forgetting, with substantial gains on hard math, self-improvement, and knowledge injection.
More from Research
- Scalable gradient attribution finds deceptively harmful training data, with a 0.6B model as monitor — PeterHndrsn · 2026-11-03
- Dynamical releases SDL-2: frontier agents make correct materials-science calls in only ~half of experiments — BenBlaiszik · 2026-11-03
- Follow-up: Astra's weight edits only worked on Qwen's visible flaw, not on Gemma 270m — teortaxesTex · 2026-11-03
- Zuckerberg's Biohub bets $1.8B on virtual cells: the bottleneck is data, not AI — HarperSCarroll · 2026-11-03
- EMBER: pretrained gene-centric foundation model predicts multigene perturbation effects — jmuiuc · 2026-11-03
- New preprint RLCE: reinforcement learning with comparative evidence for social intelligence — pliang279 · 2026-11-03