RLSD Improves Sequence-Level Reward Training

burny_tech · x · 2026-07-11

RLSD (Self-Distilled RLVR) discusses how to improve sequence-level reward training like RLVR / GRPO:

Existing GRPO averages a single sequence-level reward from a rollout across all tokens in a long chain, meaning key reasoning and filler narrative receive the same credit.
The text notes that while "on-policy self-distillation with privileged references" was previously considered to fix this, the paper proves an irreducible information gap exists: the student model chases a teacher that sees information it cannot, causing validation performance to collapse after a few training steps.
RLSD's approach separates the two: environmental rewards dictate the update direction for each token, while self-distillation signals only adjust the update magnitude within a trajectory, clipping it within a trust region similar to PPO.
This serves as a direct replacement for GRPO: requiring no auxiliary loss or separate teacher, just one extra forward pass.
On Qwen3-VL-8B, it is reported to achieve improvements across 5 multimodal reasoning benchmarks.

More from Multimodal