DCSD Splits Teacher Credit into Sign and Size, Hits 88.56 Overall on Qwen3-4B Math

Can We Trust the Teacher? Decoupled Credit Direction-Magnitude for Self-Distillation

Yugu Li, Zehong Cao, Peizhen Li, Yang Zhang, Siyi Hu, Jianglin Qiao

cs.AI

2026-09-28

DCSD splits step credit into belief-margin direction and information-gain magnitude. Qwen3-4B math overall is 88.56 (+7.16 vs Vanilla); multimodal overall is 61.14 (+7.71).

What problem this solves

RL with verifiable rewards (RLVR) scores a whole trajectory with a terminal 0/1. A correct rollout still reinforces bad or idle steps; a failed rollout gives zero credit to steps that actually helped. On-policy self-distillation (OPSD) densifies supervision by letting the same model act as a privileged teacher that sees the gold answer or a reference solution and scores token log-ratios. Direction (encourage vs penalize) and magnitude (how much) then ride on that single teacher number. The teacher can mark a harmful step as helpful, and the size of the score drifts with wording, template, and where the reference sits.

RLSD and RLCSD pin direction to the trajectory-level advantage and use distillation only to scale the update. Harmful steps in successful traces cannot get negative credit; useful steps in failed traces cannot get positive credit. That is the trade-off this paper treats as the actual problem: do not treat the teacher as an oracle, and do not give up step-level sign flips.

Method

Decoupled Credit Self-Distillation (DCSD) sets update direction and update size on the student side. The teacher only splits credit among tokens inside a step.

• Step cuts come from change-point detection on a fixed hidden layer, after generation, so decoding does not change.
• Direction comes from belief-margin probing. At each step boundary the model sees a fixed immediate-answer suffix and teacher-forced log-likelihoods over a frozen candidate set. If the correct-answer margin change ΔMk clears a threshold, the local sign is used; otherwise the method falls back to sign(Aτ). The test is whether the step moved relative mass toward the gold answer.
• Magnitude comes from marginal information gain. Representation volume is F(S)=½ log det(I+βΣhh⊤); ΔFk is how much new volume the step adds beyond the prefix. Max-normalization inside the response yields αk in [0,1]. Repeated or empty steps add little volume and get small weight.
• Token credit is At = σk · κτ · |Aτ| · αk · q{k,t}, with q a clipped teacher-student log-ratio normalized inside the step. Theorem 4 says no teacher realization can change the step's sign or its total credit.

The design skips estimating intermediate success rates, which would need a value head or continuation samples. Information gain only measures new structure in the response; correctness is the probe's job.

Results

Math uses Qwen3-4B on DAPO-17K. The metric is mean@4; Overall is sample-weighted.

DCSD Overall is 88.56, +7.16 vs Vanilla (81.40) and +8.45 vs OPSD (80.11). The abstract attributes +8.45 to the base model; the table ties that gap to OPSD. AMC23 moves from 92.50 to 95.00; AIME26 from 52.50 to 59.17. OPSD sits below GRPO on all three AIME sets.

Multimodal uses Qwen3-VL-8B-Instruct. Only DCSD was trained here; other rows are copied from the RLSD paper under the same setting.

Overall 61.14 is +7.71 vs Vanilla and +2.95 vs RLSD. WeMath jumps from 58.00 to 67.05. ZeroBench drops to 20.66, below RLSD at 24.85.

Training reward tracks GRPO and RLSD; validation accuracy is higher. Direction overrides hit about 6% of tokens, both positive in failed traces and negative in successful ones. Mean absolute advantage shrinks 1.50× (Δ=0.193). Against RLCSD, unweighted mean Pass@1 and Pass@4 rise 4.02 and 3.74 points across six math benches, with 6.3% shorter responses. AIME26 Pass@1 is 56.67 vs RLCSD 60.00.

On 90 fixed AIME24-26 rollouts, varying only privileged context, a matched Qwen3-32B is the proxy reference. Across five conditions (no answer, correct, incorrect, equivalent form, worked solution), DCSD mean top-1 disagreement is 9.82%, vs 12.12% OPSD and 10.16% RLSD.

Why it matters

If you already run on-policy self-distillation, the teacher log-ratio should not be the step advantage. It is a within-step allocator. Direction can come from the student's gold-answer margin; magnitude can come from new representation volume. For 4B-8B reasoning runs on verl, this is a credit rewrite that can keep the data and the rollout budget.

The gain is incremental. A 6% sign override means most steps still follow the trajectory sign. The more plausible source of the lift is shrinking credit on redundant steps.

Limitations

The conclusion only flags longer-horizon and interactive reasoning. The main text already lists harder constraints.

The belief probe needs the gold answer as a scoring target, so it is verifier-anchored, not an unsupervised process reward. Theorem 2 further needs readout error Ξk below the threshold; the paper states the online probe does not estimate Ξk, so an empirical threshold does not certify every override.

Information gain bounds expected squared oracle advantage; it does not recover |Ak| pointwise. Multimodal baselines were not re-run in the same job. ZeroBench's drop is blamed on visual perception, which credit calibration cannot fix. Qwen3-32B is a stronger model, not an oracle. The extracted main text has no ablation that isolates the probe, the information gain, and the teacher allocator, so the main tables do not show how much each piece contributes. Scale stops at 4B and 8B.

Terms

• OPSD: on-policy self-distillation; the same model generates as student and scores as a privileged teacher that sees the answer.
• credit direction / magnitude: sign and size of a step update; this paper refuses to take both from one teacher scalar.
• belief-margin probing: score the gold answer against competitors at step boundaries; use the margin change as the local sign.
• marginal information gain: increase in hidden-state covariance log-det, used as relative step size.
• no-crossing: forcing local credit to share the trajectory advantage's sign, so a wrong step in a correct trace cannot be penalized.

Source

• arXiv 2609.34848v1
• PDF
• Hugging Face

Related papers

• RIDAR Detects Integrity Violations in 99.7% of Research Agent Runs and Cuts Them 36.8% at Runtime
• Agents that build their own benchmarks: best matches 75% of target rankings, below the 91% ceiling
• Early-fault success goes from 0% to 100%; late-fault tokens fall from 7566.7 to 3694.4
• Self-built ARC harnesses score from 0 to 97.14; two revisions add 2.80
• ToM2Vec adds 12 Overcooked points by aligning 10 latent tokens to a partner
• T2Quant reallocates a fixed 4-bit DLM budget over denoising steps, gaining up to 6.7 points

All paper explainers