One A100, 30 minutes: JANUS lifts LLaMA-2 knowledge retention past the pretrained baseline

Past, Future, All at Once: Mitigating Stability-Plasticity Dilemma via Post-hoc JANUS Rectification

Zhilong Zheng, Letian Tao, Yujie Yang, Wei Xiong, Kehua Sheng, Bo Zhang, Jingliang Duan, Keqiang Li, Yang Guan, Shengbo Eben Li

cs.LG, cs.AI

2026-09-17

JANUS projects updates into the Jacobian null space with no retraining, lifting LLaMA-2 PiSSA knowledge from 68-80% to 100-110% of pretraining while math and code barely move.

What problem this solves

Fine-tuning on math, code, or instructions knocks down the general knowledge a model had at pretraining. TriviaQA, NQ open, and WebQS drop. That is catastrophic forgetting, and it is the stability-plasticity dilemma in operational form: keep the old skills, or make room for the new ones.

Recent fixes try to handle this inside LoRA training with subspace orthogonality. For every linear layer the update ∆W must satisfy ∆W M = 0, so its row space is orthogonal to M. M is either the layer input on old data, or a stand-in built from past parameter updates. The condition implies a zero first-order change in the old loss. It asks for something stricter. The input dimension has to hold as many mutually independent directions, all orthogonal to M, as the layer has outputs. That set exists only when din minus the rank of M is still at least dout. Stack a batch and M is often full row rank, so the allowed directions disappear. Methods that enforce the constraint by initialization also freeze LoRA's A matrix. A frozen A cuts plasticity, and it cannot sit next to PiSSA, which initializes the adapter from the weight matrix's principal components.

Parameter-space orthogonality treats the whole layer as one vector and only asks that its inner product with a single gradient be zero: Tr(∆W M) = 0. A zero trace does not require a zero update. Individual rows can move if their first-order contributions cancel. A modern layer has din times dout parameters, so such a direction is easy to find. JANUS runs the projection after fine-tuning and leaves the training run alone.

Method

Finish any fine-tune. Keep the pretrained weights and the fine-tuned weights. Replay 256 examples from the old task and, for each linear layer, stack one gradient per example into a Jacobian J. Each row is that example's gradient on every parameter in the layer. Remove from the update its component in the span of those gradients. What remains lies in the Jacobian null space and is orthogonal to all 256 gradients.

Under a first-order expansion, the loss change is that inner product. A zero inner product means the replayed examples see approximately no first-order loss change. Theorem 1 lifts the per-layer fact to the network: project every layer on its own, and the total first-order change on those examples is zero. The average gradient is the wrong target. Replacing the per-example Jacobian with the mean drops stability back to the unrectified model, because the mean smears out directions that differ across examples.

The Jacobian is only trustworthy near the point where it was computed. Multi-step adaptive rectification watches the JANUS shift, the mean cosine of the principal angles between Jacobian row spaces at the current point and at a trial point. If that cosine is at least τ, the step is kept. Otherwise the step size is multiplied by β and retried. A step shorter than 1 triggers a fresh Jacobian, and the walk continues along a contour of the old loss. Every main experiment uses τ = 0.95 and β = 0.7.

J has shape equal to the example count times the parameter count, so it is never materialized. Ghost projection uses the layer inputs and pre-activation gradients from one backward pass and reduces the projection to tensor contractions. The orientation check uses the same contractions, with no per-example backward. A rank-32 SVD then compresses the sequence axis. On that implementation, rectifying LLaMA-2-7B fits on one A100 and finishes on the order of 30 minutes, with a float16 cache under 40 GB. One profiled JANUS-P run on math took 1455 seconds: forward plus backward 4.46%, SVD 36.07%, disk I/O 28.19%.

Results

The bases are LLaMA-2-7B and LLaMA-3-8B. New tasks are GSM8k and MATH, HumanEval and MBPP, and the first turn of MTBench scored by GPT-4. Old knowledge is TriviaQA, NQ open, and WebQS. Baselines are full fine-tuning, LoRA, PiSSA, and the training-time methods CorDA, MiLoRA, and LoRA-Null. JANUS is applied to the first three checkpoints as JANUS-F, JANUS-L, and JANUS-P.

AP1 averages the knowledge benchmarks after normalizing each to the pretrained model, so 100 means parity. AP2 does the same for the new tasks against full fine-tuning. PiSSA and JANUS-P on LLaMA-2, and LoRA and JANUS-L on LLaMA-3, are reported over three seeds.

On LLaMA-2, JANUS-P moves PiSSA's AP1 from 67.55–79.71% to 100.27–109.83%.

TaskPiSSA AP1 / AP2JANUS-P AP1 / AP2
Math67.55±3.85 / 72.56±1.37106.16±4.98 / 71.54±1.49
Code79.71±2.20 / 75.94±1.24109.83±2.51 / 75.19±0.80
Instruction following79.53±2.88 / 81.80±11.94100.27±1.18 / 75.58±5.59

Math and code barely give back new-task score. Overall AP, the mean of AP1 and AP2, rises from 70.05 to 88.85 on math and from 77.83 to 92.51 on code. Instruction following does not follow that pattern: AP2 falls from 81.80±11.94 to 75.58±5.59.

On the same math setup, CorDA, MiLoRA, and LoRA-Null land at overall AP 70.59, 64.42, and 69.25. Where the underlying fine-tune itself loses a new-task benchmark to those methods, rectification does not invent the missing points.

LLaMA-3 does not get back above pretraining. JANUS-L raises LoRA's AP1 from 57.02–75.96% to 87.07–91.23%, and keeps more than 98% of LoRA's AP2 on every task: math 100.73 to 100.54, code 110.46 to 109.50, instruction following 101.85 to 100.00. Full fine-tuning is harder to repair. On LLaMA-2 math its AP1 is 39.97; JANUS-F returns it to 85.55 while AP2 stays at 97.69.

The multi-step schedule is doing real work. Setting τ to 0 collapses the procedure to one projection and, on seed 233, drops Math AP1 from 111.55 to 91.73 for JANUS-P on LLaMA-2, and from 90.13 to 69.29 for JANUS-L on LLaMA-3. After a single step, AP1 correlates with the JANUS shift at p = 0.029 and p = 0.00134 on the two models. Swap the per-example Jacobian for the average gradient and AP1 falls from 111.55 to 70.80, level with PiSSA at 71.27. Cutting the replay buffer from 256 to 64 examples drops full-finetune math AP1 from 85.55 to 63.41, while AP2 stays near 97.

An AP1 above 100 is not three knowledge sets beating the base model. For JANUS-P on LLaMA-2 math, TriviaQA is 47.31 against a pretrained 52.52, NQ is 16.05 against 18.95, and WebQS is 8.35 against 5.81. The first two remain below the base. WebQS starts from 5.81, and that small denominator pulls the average over 100.

Why it matters

The piece from Tsinghua and DiDi is a post-process. A finished checkpoint can be patched without another training run and without a penalty coefficient in the optimizer. At 7B to 8B the patch is about half an hour on one A100, with a cache under 40 GB. On math and code, normalized old-knowledge scores return to roughly the pretrained level or above it, and new-task scores stay where fine-tuning left them. Instruction following is less tidy. How high the new-task scores sit still depends on the fine-tune underneath.

Limitations

The guarantee is a local first-order approximation. On a rough loss surface, or under a large domain shift, accepted steps get short and rectification gets slow. The study covers pretraining followed by one new task, not multi-task continual learning, and only LLaMA-2-7B and LLaMA-3-8B.

The replay distribution picks which knowledge comes back. The default 256 examples come from NQ open, the same source CorDA and LoRA-Null use. Replay from TriviaQA or WebQS instead, and full-finetune math AP1 falls from 85.55 to 67.94 or 56.57. Full fine-tuning on LLaMA-3 is unstable at larger learning rates, so it uses 2e-6 to 5e-6 while the other methods use 5e-5. That column is not a matched optimizer comparison. Instruction-following training data is single-turn, and the reported score is only the first turn of MTBench. On the same full-finetune math setup, compressing the sequence axis to rank 4 drops AP1 from 85.55 at rank 32 to 75.57.

Terms

Source

What people are saying

Related papers

All paper explainers