Xiaohongshu AllSpark's ROSS relearns from past rollouts, adding up to 4.2 points on SWE-bench

小红书技术REDtech · wechat · 2026-10-20

Xiaohongshu's AllSpark team published ROSS (Relearning from Self-Generated Rollouts through Selective Supervision), a method that mines historical training rollouts for additional learning signal after RL or on-policy distillation has finished. Code is open-sourced.

Key insights

Historical rollouts remain highly compatible with the current policy: on math problems, their average NLL under the current model (0.120) matches current-policy outputs, far below external teacher outputs (0.520)
Compatibility ≠ mastery: on 98 previously-solved hard problems, the final model's pass@1 was only 37.7% (pass@32: 99.0%), showing many effective behaviors were never consolidated
Naive SFT on all successful rollouts actually hurts performance; selective supervision is essential

Method

Filter successful trajectories with task verifiers (answer checks, unit tests, or environment signals)
Use an LLM judge (GLM-5.2 high-thinking) to audit and locate imitable spans in the original text; discard samples without reliable targets
Masked SFT: keep full context, compute loss only on selected model-generated tokens — past mistakes aren't imitated but remain as context for understanding later corrections

Results (Qwen3.6-35B-A3B)

Single-turn RL: math 75.19%→76.98%, code 42.09%→44.21%
Multi-teacher on-policy distillation: 58.40%→62.20% (+3.80), gains across all six benchmarks
Agentic RL: SWE-bench Verified 64.20%→68.40% (+4.20)

Original post →

Xiaohongshu AllSpark's ROSS relearns from past rollouts, adding up to 4.2 points on SWE-bench

More from coding & agent