SFT Conflicts, RL Coexists: A Theoretical and Empirical Analysis of Multi-Task Learning for LLMs
Kejian Zhu, Zhuoran Jin, Shangqing Tu, Hongbang Yuan, Yushi Bai, Kang Liu, Juanzi Li, Jun Zhao
cs.CL, cs.LG
2026-08-04
Multi-task SFT drops 23.1% but RL rises 24.9% on DeepSeek-R1-Distill-Qwen. RL updates are sparse and near-orthogonal across tasks, so task interference is tiny. Parallel-RL trains tasks independently and merges them, retaining ~103% of single-task RL.
When post-training LLMs on multiple tasks, two paradigms dominate: supervised fine-tuning (SFT) and reinforcement learning (RL). A persistent pain point is catastrophic forgetting, where training a later task erases the previous one's competence. This paper asks why sequential multi-task SFT conflicts badly while sequential multi-task RL coexists stably.
The authors set up a controlled comparison on DeepSeek-R1-Distill-Qwen-1.5B over the same task suite (math, science, logic, code). Multi-stage SFT ends up 23.1% below the base model on average, with logic alone dropping 22 points; multi-stage RL ends up 24.9% above the base, and every task improves. The gap is wide enough to chase to the bottom.
They move the analysis to the parameter level, give a theoretical account, and finally propose a training paradigm.
At the parameter level, train one step of the same model on each task separately and inspect the update ΔW. RL updates have two signatures: the L2 norm of ΔW is about 3×10⁻² for RL versus about 7.4 for SFT, two orders of magnitude apart, and RL updates are highly sparse. The decisive signal is cross-task cosine similarity: SFT updates across tasks sit around 10⁻¹ (clearly overlapping directions), while RL updates sit around 10⁻³ (nearly orthogonal). Directions that do not overlap cannot easily step on each other.
For the theoretical account, writing the upper bound on cross-task gradient interference reveals that the two paradigms are limited by different quantities. SFT interference is norm-limited: the bound scales with the gradient magnitude itself, so denser, larger gradients interfere more, matching SFT's large-norm, overlapping behavior. RL interference is variance-limited: the bound is set by the gradient variance left after advantage normalization. Because RL advantages are zero-sum and cancel within a batch, the mean gradient direction drops out and only intra-group residuals remain; those residuals have small variance and, as sparse zero-mean vectors in high dimensions, are close to orthogonal with high probability. Interference is squeezed small.
Parallel-RL follows directly. Since different tasks' RL updates are approximately orthogonal, ⟨ΔWᵢ, ΔWⱼ⟩ is about zero, so training tasks independently and adding the updates back approximates sequential training while running in parallel. The merge has three variants: direct sum, sparse merging via TIES or SVD, or a short adaptation pass with 5% of the original data after merging. The adapted variant works best.
Base models are DeepSeek-R1-Distill-Qwen-1.5B and 7B, over MATH500/AIME2025, MMLU/GPQA, Knights and Knaves, and LiveCodeBench.
| Paradigm | 1.5B avg (vs base) |
| Multi-stage SFT | -23.1% |
| Multi-stage RL | +24.9% |
| Mixed-data RL | +12.6% |
Single-task transfer: SFT gains 4.0% on the target task and loses 5.1% on untrained tasks; RL gains 6.8% on the target and 2.3% on untrained tasks (positive transfer). The ΔW L2 norm is about 3×10⁻² for RL versus about 7.4 for SFT.
Parallel-RL (1.5B, GRPO) retains 94.2% of single-task RL performance with a plain sum, 97.4% with TIES merging, and 102.8% with the adapted variant (102.4% on 7B). In an ablation, removing one task's update drops that task by 7.1% while nudging the others up 0.6%, confirming the capabilities are decoupled.
The paper turns an engineering intuition (RL tolerates multi-task training better than SFT) into an explained mechanism: orthogonality comes from the zero-sum structure of the advantage, not from luck. The practical guidance is direct. When tasks fight under SFT, switch to RL or split them into parallel tasks and merge, dodging sequential forgetting. Parallel-RL turns multi-task training from a queue into a parallel workload, saving wall-clock time and letting tasks be added or dropped flexibly.
Be honest about the boundary: mixed-data RL (multiple tasks in one batch) gains only 12.6% versus 24.9% for multi-stage, which the authors attribute to gradient imbalance across tasks but call outside the focus of the work. The orthogonality holds when tasks are separated; mixing them into one batch changes the mechanism.
The authors' self-stated limits are thin: they note the weaker mixed-data RL gain and that one must judge which tasks parallelize well. The sharper questions are elsewhere. The orthogonality argument rests on advantage-zero-sum algorithms like GRPO; under reward shapes that are not zero-sum, or RL variants without advantage normalization, whether the variance-limited bound still holds is unclear. The paper extends the theory to PPO with a dedicated bound theorem, but the empirical validation concentrates on GRPO. All numbers come from 1.5B and 7B distilled models; consistency at larger scale or on non-distilled models is unverified.