ROSS: Relearning from Self-Generated Rollouts through Selective Supervision
Zhiwei Zhang, Huayu Deng, Fei Zhao, Jiayan Fu, Bin Liang, Kam-Fai Wong, Mu Chuan
cs.LG, cs.AI
2026-09-29
ROSS applies masked SFT to useful spans of historical self-rollouts. On Qwen3.6-35B-A3B, MOPD average goes 58.40% to 62.20% and SWE-bench Verified 64.20% to 68.40%.
RL and on-policy distillation both leave behind trajectories the model sampled itself. In on-policy distillation the student acts first and the teacher only supervises states the student actually visited. Once training moves on, those traces are usually thrown out as stale. The later policy's sampling distribution has shifted, so a behavior found early may still be legal and still useful, yet show up unreliably in the current checkpoint.
On the 100 hardest training problems that earlier checkpoints had solved, the final policy's pass@1 is 37.7%, and 74 of 98 problems sit at or below 50% success. pass@32 on the same set is 99.0%. The behavior is still in the support; it is not consolidated. Token NLL of those historical replies under the current policy sits near on-policy samples and well below an external teacher, GLM-5.2. Compatible, not yet reliable.
A correct final answer does not make every step worth copying. Successful traces still mix mid-solution errors, abandoned attempts, and exploratory detours. Outcome-level SFT writes all of that into the gradient.
ROSS starts from the final checkpoint of the original run and does one offline masked SFT pass on saved history. No new policy rollouts. Selection is two-level.
The full original prefix stays in context, mistakes and all. Loss lands only on tokens with mask 1. Single-turn traces use positive-span extraction; agent traces use defect localization; both compile to the same token mask.
ROSS w/o mask uses the identical example set but supervises every policy-generated token, so trajectory filtering and span selection can be scored separately.
The main model is Qwen3.6-35B-A3B. Single-turn runs are no-thinking, three SFT epochs. Baselines: the public Base checkpoint, Upstream (end of the original RL or MOPD run), Continued training for two extra epochs or 200 extra MOPD steps, and Positive-Rollout SFT on every verifier-positive trace. Agentic runs train on OpenSWE via a Codex CLI agent in Harbor, then evaluate on SWE-bench Verified with a frozen scaffold.
| Setting | Metric | Upstream | Continued | Full-trace SFT | ROSS |
| Math RL | Math Avg. | 75.19 | 75.43 | 75.99 | 76.98 |
| Code RL | Code Avg. | 42.09 | 42.31 | 40.84 | 44.21 |
| MOPD | six-benchmark avg | 58.40 | 59.16 | 57.70 | 62.20 |
| Agentic RL | SWE-bench Verified | 64.20 | not reported | 65.20 | 68.40 |
Full-trace SFT hurts MOPD: 57.70 versus Upstream 58.40, and IFBench falls from 44.58 to 41.49. ROSS w/o mask recovers the average to 58.49. Full ROSS reaches 62.20, 3.71 points above the unmasked variant. Most of the gain is the token mask.
The same traces and masks, started from Base rather than Upstream, land in the same band: Math Avg. 77.09 versus 76.98 after Math RL. The behaviors transfer without inheriting the original RL parameter updates.
Masked content does not fade as the policy gets more certain. Over the first 100 RL rollout steps, entropy falls, but the target exclusion rate among kept traces only moves from 11.60% to 9.27% on Math and rises from 22.65% to 28.07% on Code. Among 1,185 classified traces, Math is mostly error-then-correction (72.7% early, 68.3% late); Code is mostly redundant exploration (67.9% to 76.6%). A separate 4B Math RL run in the appendix grows a repetitive-answer mode; ROSS lifts Avg. Math from 52.05% to 63.65% and suppresses that mode.
Off-domain scores do not collapse. After Math RL, Avg. Code goes 41.21 to 44.24 and IFBench 33.30 to 34.60; after Code RL, Avg. Math goes 74.19 to 75.03. BFCL moves 44.12 to 45.62 and 46.25 to 49.38.
Successful traces sitting in post-training logs are already near-on-policy data. Teams that already dump rollouts can add one offline SFT stage at the final checkpoint and skip another sampling round. Relative to another RL pass, this reuses logs whose sampling cost is already paid. The dump policy has to be set up front: intermediate successful traces cannot be rebuilt later if they were never saved. Copying whole successful traces is a net loss on MOPD; if span filtering is not available, skip this stage. The recipe is close to SWE-Prime's full-context selected-span SFT. What is new is the data source: behaviors an intermediate policy found and a later checkpoint no longer emits reliably.
If historical rollouts were never saved, or the task has no trustworthy verifier, the recipe does not apply. On this 35B model the lifts are about 2 to 4 points. That is consolidation of behaviors the run already discovered.
The main text has no Limitations section. Almost every headline number is Qwen3.6-35B-A3B; the 4B result is math only. Span labels come from GLM-5.2, with no human agreement figure and no annotation cost in the main paper. The Continued baseline is only two extra epochs or 200 steps, a weak stand-in for just training longer. Single-turn eval is no-thinking and should not be mixed with the thinking-mode agent numbers. On AIME 25, ROSS scores 76.46 against 76.67 without the mask, so the mask does not help every slice.