RISE: Recursive Improvement via Self-Extrapolating Policy Distillation
Yang Li, Semih Yavuz, Shafiq Joty
cs.AI
2026-09-04
RISE extrapolates the model's own RLVR trajectory in logits or weights to build a teacher. On OLMo-7B, math average rises from GRPO's 47.6 to 56.4 with no extra sampling.
RLVR assigns one outcome to an entire response, so every token shares the same advantage. On-policy distillation can put a full next-token distribution at every position, but the teacher is the bottleneck. An external stronger model guesses badly once the student leaves its prefix distribution. A self-teacher that is shown the answer in context is capped by in-context learning. Most patches gate or mix a bad teacher. None asks what the teacher should be.
RISE's answer is a future self, built by extrapolating the training path.
Each iteration runs RLVR from θn to θ'n+1. The displacement θ'n+1 − θanchor is the recent improvement direction. Extrapolating with β>1 builds πfuture; the student is then pushed toward that teacher's token distribution, yielding θn+1. At β=1 the teacher is the current checkpoint and distillation is empty.
Two instantiations. In weight space, θfuture = θanchor + β(θ'n+1 − θanchor), task arithmetic past the current point, with a forward pass through those weights during OPD. In logit space the same rule is applied to next-token log-probs, a geometric mixture πanchor^(1−β) · π'^(β), which is the first-order picture of the weight-space move. In practice a top-100 plus a tail bucket is distilled with Jensen–Shannon divergence so reverse KL cannot explode.
β decays linearly from 1.2 to 1, because overshoot gets cheaper to trigger near optimality. Qwen runs use an EMA anchor (η=0.1); OLMo uses the previous checkpoint. RLVR rollouts are reused, so there is no extra sampling. Wall time is about 1.3–1.6× GRPO.
Both instantiations beat GRPO and privileged-context self-distillation on math.
| Setup | GRPO math avg | Best RISE | AIME24 |
| Qwen3-8B | 60.0 | 62.7 (weight) | 54.4 → 58.1 |
| Qwen3-1.7B | 45.4 | 50.2 (logit) | 30.0 → 36.3 |
| OLMo3-7B | 47.6 | 56.4 (logit) | 30.2 → 46.9 |
GRPO+SDPO lands below pure GRPO on both Qwen models. Out-of-domain GPQA / IFEval / MMLU-Pro do not drop; Qwen3-8B OOD average moves from 70.6 to 72.0. On mixed math plus STEM, weight-space RISE reaches math 44.8 vs GRPO 40.2 and STEM 47.5 vs 45.5. On code, final accuracy is similar, but HumanEval+ hits GRPO's last score at step 50 instead of 90. On agents, Qwen2.5-3B goes from 75.0 to 84.4 on ALFWorld and from 63.3 to 74.2 WebShop Acc.
Remove RLVR and extrapolate along self-distillation alone: MATH-500 falls to 2.4% within 60 steps and length hits 8K. Remove OPD and adopt θfuture directly: math average barely moves (8B: 60.0 → 60.3). Extrapolation supplies a direction; distillation is what turns it into a stable gain. β=2 still helps early and is catastrophic by step 100, which is why the schedule decays.
OPD no longer needs a stronger external model, or an answer stuffed into the teacher's context. Sparse outcome reward keeps the direction honest; the extrapolated teacher spreads that same signal across tokens. For a team already running GRPO this is an extra loss on the same rollouts, with the largest lift on contest math and about ten points on the agent suites.
No new task information enters. If the reward can be hacked, extrapolation amplifies the hack.
The method needs a low-dimensional, roughly linear trajectory at modest β. Three directions explain about 87% of variance in their runs; there is still no online detector for when extrapolation goes stale. Code benchmarks saturate quickly, so the lift is smaller than on math. The EMA anchor helps Qwen and hurts OLMo (56.4 vs 50.1 at η=1), so the anchor rule is not universal. A head-to-head with ExOPD, which needs a stronger external teacher, is not possible by design.