SFA paper: simple parameter averaging beats replay in continual learning with 36% less training
jefrankle · x · 2026-11-20
Updated preprint "Forget About It" (arXiv:2501.05559) by Anat Kleiman, Gintare Karolina Dziugaite, Jonathan Frankle, Sham Kakade and Mansheej Paul introduces SFA (Sequential Fine-tuning with Averaging). On Qwen3-14B adapter continual learning, fixed configurations of SFA and replay-free Lookahead beat tested replay and adaptive-merging setups in mean final accuracy while using up to 36.3% less task training time. Across language and vision tasks they compete with more complex continual-learning methods, suggesting simple in-training interpolation deserves a try before elaborate forgetting-mitigation mechanisms.
More from Research
- MacMimic trains humanoid motion controllers in 22 minutes on a MacBook Pro — KyleMorgenstein · 2026-11-20
- alphaXiv launches preprint server for AI agents: 936 agents wrote 936 papers in 12 weeks — SRSchmidgall · 2026-11-20
- CD19 CAR T shows immune reboot in small MS trial, Cell reports — EricTopol · 2026-11-20
- Brynjolfsson's GDP-B project proposes new measures of digital-economy well-being — erikbryn · 2026-11-20
- DNeo diffusion drafter speeds up LLM inference by up to 30% — Scobleizer · 2026-11-20
- Open-source cancer model from JHU and Stanford hits every tumor across 34 public CTs — MaziyarPanahi · 2026-11-20