Tsinghua's JANUS Fixes Catastrophic Forgetting Post-Hoc, Restoring Old Knowledge Beyond Baseline

机器之心 · wechat · 2026-11-22

A joint Tsinghua–DiDi autonomous driving study proposes JANUS (accepted to NeurIPS 2026), which tackles catastrophic forgetting in LLM fine-tuning via post-hoc rectification instead of training-time constraints.

Core idea: the classic subspace orthogonality constraint (ΔW·M=0) is sufficient but unnecessarily strict; JANUS relaxes it to the necessary-and-sufficient condition Tr(ΔW·M)=0, projecting parameter updates onto directions where old-task loss is first-order invariant via Jacobian null-space projection. Since the direction is only locally valid, it uses multi-step adaptive rectification—small repeated projections with recomputed Jacobians and an adaptive step size—plus Ghost operations and SVD compression: a 7B model needs just one A100 for 30 minutes.

Results: on LLaMA-2-7B, AP₁ (old-knowledge retention) rose from 67.55% to 106.16% on math and 79.71% to 109.83% on code, with only 1 point drop in new-task ability; gains hold across base models, tasks, and fine-tuning methods. Paper and code are public.

Original post →

More from Research

Research channel →