SPACE: undo part of SFT with zero retraining to boost LLM scores both in-domain and out-of-domain
kaiwei_chang · x · 2026-10-21
Jiachen Gu's team introduces SPACE, a post-hoc calibration method addressing the problem that fine-tuning improves a target skill while weakening others.
- Method: using only pre- and post-SFT checkpoints, SPACE applies geometric calibration over the principal subspace of the pre-SFT weights — restoring pre-SFT capabilities in the principal subspace while preserving post-SFT skills in the orthogonal complement. No extra training, no data; one step of code.
- Results (from the paper's radar charts): on Qwen2.5-7B, average gains of +2.85 in-domain (ID) and +2.26 out-of-domain (OOD); on Qwen3-14B-Base, +2.41 ID and +1.52 OOD — across MATH500, AIME24/25, AMC23, GPQA-D, MMLU-Pro, IFEval, HaluEval and more. SPACE mitigates OOD capability regression while further improving in-domain performance over the SFT checkpoint.
More from Research
- TokenGate: learned token pruning speeds up LLM prefill up to 11.3x — Liana Patel · 2026-10-21
- Terence Tao proposes AI-assisted challenge to revisit a classical Gauss problem — robleclerc · 2026-10-21
- OSWorld-Science benchmark: 12 VLMs average just 34.3% on real scientific software tasks — jiqizhixin · 2026-10-21
- Experiments show Qwen3 can compile arbitrary patterns into layer-by-layer steps with plan training — Sauers_ · 2026-10-21
- 'Functional Welfare' paper shows repeated rejections degrade Gemma 3 27B behavior — repligate · 2026-10-21
- Someone actually trained a model with zig-zag activations, code and weights open-sourced — Sauers_ · 2026-10-21