Tsinghua's JANUS Fixes Catastrophic Forgetting Post-Hoc, Restoring Old Knowledge Beyond Baseline
机器之心 · wechat · 2026-11-22
A joint Tsinghua–DiDi autonomous driving study proposes JANUS (accepted to NeurIPS 2026), which tackles catastrophic forgetting in LLM fine-tuning via post-hoc rectification instead of training-time constraints.
Core idea: the classic subspace orthogonality constraint (ΔW·M=0) is sufficient but unnecessarily strict; JANUS relaxes it to the necessary-and-sufficient condition Tr(ΔW·M)=0, projecting parameter updates onto directions where old-task loss is first-order invariant via Jacobian null-space projection. Since the direction is only locally valid, it uses multi-step adaptive rectification—small repeated projections with recomputed Jacobians and an adaptive step size—plus Ghost operations and SVD compression: a 7B model needs just one A100 for 30 minutes.
Results: on LLaMA-2-7B, AP₁ (old-knowledge retention) rose from 67.55% to 106.16% on math and 79.71% to 109.83% on code, with only 1 point drop in new-task ability; gains hold across base models, tasks, and fine-tuning methods. Paper and code are public.
More from Research
- Stanford CS312 teaches deep learning by experiment prediction; suggested as synthetic-data seed for frontier labs — jasondeanlee · 2026-11-22
- John Langford weighs in on FBT note: fixed-point convergence and quantized caching — JohnCLangford · 2026-11-22
- Fields Medalist Bao Chau Ngo on AI proof of Riemann Hypothesis: treat AI output as data — Dr_Atoosa · 2026-11-22
- AI-written six-page proof of Deligne's bound lands, checked only by Claude models — Singularitarian · 2026-11-22
- AI math pace outruns expert timelines: two Millennium Prize-level problems now reportedly solved — haider1 · 2026-11-22
- Darmok AI solves all 7 Times Crossword Championship puzzles, beats human champ by 3m13s — rohanpaul_ai · 2026-11-22