Alibaba I-SDPO: distill only all-incorrect GRPO groups, SciKnowEval 56.67% to 70.31%

I-SDPO: Instance-Level Adaptive Self-Distillation Policy Optimization

Yubo Zhang, Xinhong Ma, Zezhong Tan, Ziqiang Dong

cs.LG, cs.CL

2026-08-13

I-SDPO distills only when every rollout sample fails and keeps GRPO otherwise, raising Qwen3-8B SciKnowEval mean@16 accuracy from 56.67% to 70.31%.

What problem this solves

GRPO has become the default RL algorithm for LLM reasoning training: sample a group of responses per prompt, subtract group-relative advantages, skip the critic. Its blind spot is the all-incorrect group. On a hard problem where all 16 samples are wrong, rewards are nearly identical, advantages collapse toward zero, and the gradient vanishes. The paper quantifies this: at a per-sample success rate of 0.1 with K=16, roughly 18.5% of groups produce no relative signal at all. The weaker the policy and the harder the problem, the less signal GRPO gets, exactly where learning is most needed.

Privileged self-distillation fills the gap: a teacher that sees the ground-truth solution assigns token-level soft targets along the student's own trajectories. The signal is dense and low-variance, but the teacher is biased. It is an EMA copy of the student and inherits the student's systematic errors; it favors one demonstrated solution style while the reward may accept many valid paths. Distilling throughout training keeps pulling toward a teacher-specific distribution after reward-bearing trajectories become available.

The tension is that distillation helps early and hurts late, and "early or late" is not a global clock. It varies per prompt: an easy prompt may yield a success in the first batch, a hard one may stay all-incorrect until the end of training.

Method

I-SDPO lets each rollout group vote on whether to trust the teacher. One routing decision per instance, shared across the whole group:

• Sample K=16 responses per prompt, compute rewards for all.
• If any response is correct, the entire group goes to GRPO. A mixed group keeps its full internal contrast; the comparison between 15 wrong answers and 1 right one is exactly the evidence reward learning needs.
• If all responses are wrong and ground truth exists, the whole group goes to SDPO distillation. Within-group comparison carries no information here, so the teacher's dense token supervision takes over.
• If all are wrong and no ground truth exists, fall back to GRPO (effectively no gradient).

The teacher is not an external model but an EMA copy of the student (update rate 0.05, mean lag of about 19 updates). Its privilege is in the input: the ground-truth solution is prepended to the prompt before it scores the student's generated tokens position by position. The distillation loss interpolates forward and reverse KL at α=0.5, with entropy-aware token weighting: positions where the teacher distribution has low entropy (high confidence) get larger weights. The authors are explicit that this is a confidence proxy only and cannot fix a confidently wrong teacher.

Two theoretical results back the design. Proposition 1: when the teacher target sits a bias b away from the reward optimum, a non-vanishing distillation weight λ parks the parameters at λ/(1+λ)·b from the optimum, with excess reward loss about ½[λ/(1+λ)]²‖b‖². Persistent distillation has a bias floor. Proposition 2: if the per-sample success probability is non-decreasing and the K rollouts are conditionally independent, the all-incorrect probability (1−pt)^K is non-increasing. Routing drives the effective λ toward zero as capability grows; the teacher withdraws without a hand-designed schedule.

This self-annealing is per prompt: at the same training step, easy prompts have long returned to GRPO while hard ones are still distilling. The sample-level alternative (SRPO) routes each incorrect sample individually, which dismantles the mixed group's contrast: in a 15-wrong-1-right group, SRPO converts 15 pieces of negative evidence into imitation targets.

Results

SciKnowEval, four scientific domains, Qwen3-8B, 2 epochs, mean@16 accuracy:

• +13.64 on average over GRPO, with the largest domain gain of +18.24 on chemistry. Biology is the sharpest case: GRPO sits at 32.12 while every distillation-based method clears 44, consistent with biology having the densest all-incorrect groups and the most to gain from dense supervision.
• Pure distillation beats GRPO by 9.07 but trails I-SDPO by 4.57: distilling helps, distilling indiscriminately is not the best final objective.
• +4.30 on average over sample-level SRPO, including +5.98 on biology and +7.19 on physics. Routing granularity itself contributes.

Training dynamics match the self-annealing account: on biology, the GRPO-routed fraction rises from about 0.65 at step 50 to about 0.90 at step 350, while the all-wrong fraction falls from 0.22 to 0.03. In ablation, the balanced α=0.5 KL mix wins in all four domains, averaging 68.16% versus 60.39% for forward KL alone and 61.63% for reverse KL alone.

Why it matters

All-incorrect groups are not a cold-start-only problem. A harder benchmark, a smaller base model, or a stricter reward checker pushes their share right up, and GRPO spins its wheels in those settings. I-SDPO offers a patch with almost no extra hyperparameters: the routing criterion is "does the group contain one correct answer", binary and observable, no gradient-alignment estimation needed; the teacher is a free EMA copy with no inference-side deployment cost. It replaces a hand-tuned distillation schedule with the (1−p)^K curve, which adapts per prompt.

For anyone doing RL post-training, this is the "distillation and RL each cover a phase" intuition made mechanical: switch on capability evidence per prompt, not on training step. The SRPO comparison shows granularity matters, and that negative samples in a mixed group are valuable evidence, not imitation material.

For calibration on where the gain comes from: pure distillation already captures 9 of the 13 points. I-SDPO's contribution is extending that to 13.6 while avoiding the late-stage drag, so the novelty is in the scheduling, not the distillation.

Limitations

The authors list two: only Qwen3-8B was tested, and only SciKnowEval. Stronger base policies produce fewer all-incorrect groups, so the patching value shrinks; math or open-ended reasoning tasks have different reward structure and solution multiplicity.

Further concerns from reading the paper:

• The routing rule depends on ground truth being available for the prompt. Science eval problems have verifiable answers; in settings with only a reward function and no reference solution, the SDPO branch is unusable and the method degrades to GRPO. The paper handles this only as an algorithmic fallback and never discusses coverage.
• mean@16 and the training reward are drawn from the same source: SciKnowEval serves as both training data and evaluation, so the main table measures in-benchmark fit. No external held-out benchmark is reported.
• Propositions 1 and 2 are explanatory models under local quadratic approximation and conditional independence; the authors state they establish no global convergence. Observed dynamics being consistent with the account is not proof of the mechanism.
• Ablation curves are single runs; the authors say they compare final values and treat trajectories descriptively. In the top-k ablation, I-SDPO averages differ by only 0.45 between k=40 and k=100 (69.86 vs 70.31), a gap a single run may not resolve.

Terms

• GRPO: Group Relative Policy Optimization; samples a group of responses per prompt and derives advantages from within-group reward differences, removing the critic model.
• privileged self-distillation: a teacher that sees the ground-truth solution during training (not inference) and assigns token-level soft targets on student trajectories.
• all-incorrect group (degenerate gradient): every sample in a rollout group is wrong, rewards show no spread, advantages collapse to zero, and the GRPO gradient vanishes.
• EMA teacher: a teacher copy whose parameters track the student via exponential moving average, giving smoothed but lagged target distributions.
• mean@16: average accuracy over 16 sampled responses per test problem, capturing both capability and consistency.
• LUPI: Learning Using Privileged Information; extra information available at training time but not at inference.

Source

• arXiv 2608.12957v1
• PDF
• Hugging Face

What people are saying

• Alibaba proposes I-SDPO to fix degenerate gradient problem in GRPO heghbalz

Related papers

• Yale: a coverage-SNR threshold decides SFT vs ICL, and GPU congestion can reverse it
• Penn's SSV Cuts Strong Verifier Calls 46% on Sudoku While Matching Oracle Accuracy
• Train grokking 10 million steps further and test accuracy falls back to 50%
• A 109M orbital transformer hits R² 0.9999, then recovers five fake gravity laws
• CoT monitors cannot be sound: Harvard on linguistic illegibility in LLM security
• Adam adapts per-parameter steps from first and second moments, α=0.001

All paper explainers