SFA paper: simple parameter averaging beats replay in continual learning with 36% less training

jefrankle · x · 2026-11-20

Updated preprint "Forget About It" (arXiv:2501.05559) by Anat Kleiman, Gintare Karolina Dziugaite, Jonathan Frankle, Sham Kakade and Mansheej Paul introduces SFA (Sequential Fine-tuning with Averaging). On Qwen3-14B adapter continual learning, fixed configurations of SFA and replay-free Lookahead beat tested replay and adaptive-merging setups in mean final accuracy while using up to 36.3% less task training time. Across language and vision tasks they compete with more complex continual-learning methods, suggesting simple in-training interpolation deserves a try before elaborate forgetting-mitigation mechanisms.

Related event: Harvard Paper: Simple Parameter Averaging Beats Replay in Continual Learning(2 posts)→

Original post →

More from Research

Research channel →