Subspace Tuning curbs LLM catastrophic forgetting by freezing principal singular directions
scottniekum · x · 2026-10-23
A new arXiv paper introduces Subspace Tuning (SST), a simple method that restricts each fine-tuning weight update to the bottom singular subspace, leaving principal singular directions untouched.
- Motivated by the finding that SFT concentrates updates along principal directions (causing forgetting), while on-policy methods like RL avoid them and forget less.
- SST is optimizer-agnostic and composes with SFT and self-distillation, combining tight subspace protection, dense gradient estimation, and a refresh mechanism to beat prior orthogonal fine-tuning methods.
- Protecting 99% of the weight subspace substantially improves retention while keeping strong target-task performance; the effect scales with model/data size, nearly eliminating forgetting in large models.
- Benchmark plot shows SST (stars) achieving similar new-task gains as SFT, SFT+Muon, and SDFT with far smaller losses in prior capabilities.
More from Research
- Dev trains boolean-logic LM with microsecond forward passes on a single CPU thread — cephaloform · 2026-10-23
- AdaDex turns seconds of human hand motion into generalizable dexterous robot skills — mangahomanga · 2026-10-23
- ReAPO paper: conditioning retries on failures beats GRPO in RLVR reasoning training — jasondeanlee · 2026-10-23
- Quantum-JEPA: first JEPA world model deployable on real quantum circuits (QPU) — DrMorganLevine · 2026-10-23
- Omnii Genome: Single Generalist Model Outperforms AlphaGenome Across All Benchmarks — JosephJacks_ · 2026-10-23
- Domingos Cites New Result Claiming RL Is 'Basically Useless' vs Supervised Learning — pmddomingos · 2026-10-23