Towards Continual Learning: Mitigating Forgetting With Subspace Tuning
Arnav Chavan, Kusha Sareen, Rishabh Tiwari, Abhranil Chandra, Johan Obando-Ceron, Kurt Keutzer, Scott Niekum, Inderjit S Dhillon, Devvrit Khatri
cs.LG, cs.AI
2026-10-16
Project every optimizer step onto the bottom 1% of each weight matrix's spectrum: forgetting drops from up to 57 points to ~1 while target-task accuracy holds or improves.
Fine-tuning is the standard path for specializing an LLM, and catastrophic forgetting is its standard tax. One round of plain SFT on financial data drops Qwen3-8B's five-benchmark general average by 57 points; GSM8K falls from 94.8 to 43.9 and HumanEval from 91.5 to 15.9. Pre-training data is unavailable, and post-hoc repair (ReInvoke) recovers only part of the loss while eating into the new task's gains.
Recent work ties forgetting to where the update lands in weight space. RL and on-policy distillation avoid the principal singular directions of weight matrices and forget less; SFT concentrates its updates exactly there. That suggests a direct hypothesis: freeze the principal subspace, keep the old capabilities, and let the rest of a heavily over-parameterized model learn the new task.
Subspace Tuning (SST) in one sentence: take the SVD of every linear layer, keep the top 99% of singular directions fixed, and project each optimizer step onto the remaining 1% free tail before it reaches the weights, via Π(ΔW) = (I − UkUk⊤)ΔW(I − VkVk⊤). The two-sided projection strips components that read from or write to protected directions.
Three design choices separate it from prior work:
It covers the 252 attention and MLP linear matrices; embeddings and layer norms are untouched. The optimizer itself is never modified, so Adam and Muon both compose. No extra parameters, no inference cost; on Qwen3-8B the projection adds 1.5% per step, and R10 refresh adds 37.6% (4.7% once the SVD is sharded across 8 GPUs).
Main setup: Qwen3-8B (thinking mode, traces with explicit reasoning) fine-tuned on four domains; prior capability averages GPQA-Diamond, MMLU-Pro, GSM8K, HumanEval, and IFEval.
| Method (Science) | New-task acc | Prior capability Δ |
| SFT | 77.6 | −25.5 |
| SFT+SST | 80.1 | −1.0 |
| SFT+SST+R10 | 84.0 | −0.8 |
| SDFT | 88.1 | −24.0 |
| SDFT+SST | 82.9 | −1.5 |
FinQA is the harshest case: SFT loses 57.3 points of prior capability while SFT+SST gains 1.3, with new-task accuracy up from 72.3 to 73.0. Pathology and Medicine follow the same pattern (SFT down 45 points, SST down 1-4).
Scaling is the strongest result. From 1.7B to 14B, SFT loses 30-40 points at every size while SST's forgetting shrinks from −8.0 at 1.7B to −0.9 at 14B (−1.0 at 32B over two epochs), and SST's new-task accuracy beats SFT at every size (88.1 vs 78.0 at 14B). Under a 45k-example budget, with either a fresh data stream or 9k examples repeated five epochs, SFT drifts steadily farther while SST stays flat.
SST also stays 2-4× closer to the base model in KL on WikiText-2, which surfaces in behaviors accuracy misses: fine-tuning Olmo-3-7B-Think on answer-only medical data, SFT collapses the reasoning length and SDFT starts fabricating reference solutions; SST keeps both in check. In a three-stage chain (FinQA → Pathology → Science), SST accumulates each domain, while SFT drops untrained domains by 18-28 points. Against the closest subspace baselines under their official settings, Sculpting and OPLoRA lose 26 and 18 points of prior capability on average where no-refresh SST loses 1.4, at comparable in-domain accuracy. Reverse-KL regularization toward the base model does not reproduce the trade-off: weak penalties change nothing, strong ones kill task learning without restoring capability.
This is engineered as a drop-in default: no optimizer changes, no new parameters, single-digit overhead, composable with SFT, SDFT, and Muon. The scaling trend runs with the industry's direction of travel: the larger the model, the less forgetting, essentially flat from 8B. Sequential fine-tuning becomes cumulative, and independently trained SST experts merge with far less interference (three-expert merge: −23.2 points for SFT vs −11.0 for SST). Forgetting compressed from tens of points to one or two is past the incremental-improvement range.
The authors flag three: the optimal learning rate may differ from unconstrained SFT and the full frontier is untuned; hundreds-of-billions parameter models and longer task sequences are untested (32B is the ceiling here); fixed-interval refresh could be replaced by update-aware scheduling.
Open questions from the reading: the Science run on Qwen2.5-7B-Instruct is the paper's only setting forced to loosen r from 0.99 to 0.90, so the 1% free tail is not always enough. R10 refresh costs 37.6% per step without SVD sharding. Refresh protects a sequence of updated subspaces rather than the original pretrained one, so protection relative to the base accumulates drift over long chains, and only three stages were tested. Experiments concentrate on the Qwen3 family plus one Olmo model, mostly medical and finance QA. In the learning-rate sweep, SFT attains the highest single in-domain score at the lowest LR; SST's edge is stability and retention, not universal dominance. The claim that RL forgets less is cited from other work; no direct RL arm is run.