SIS: Turning Off-Policy Tokens Back to On-Policy

青稞AI · wechat · 2026-07-11

Core Problem

The article discusses the off-policy problem in LLM RL post-training: strict on-policy sampling is theoretically most stable but requires re-rollout every step, which is too costly; engineering often uses asynchronous rollout and sample reuse, causing training data to come from a 'slower' old policy, leading to distribution shift, biased gradients, and training instability.

Limitations of Traditional Approaches

Mainstream methods usually rely on importance sampling for correction, but in long sequences the importance ratio of the whole sequence can explode or decay exponentially, so engineering often degenerates to token-level approximation combined with various clipping techniques (e.g., GRPO clip, DAPO clip-higher, GSPO, etc.) to suppress extreme ratios. The cost: stability improves, but signals from many heavily shifted tokens are also cut off.

New Method: SIS

The authors propose Selective Importance Sampling (SIS), which uses token-level Rejection Sampling:

For each token sampled from the old policy, accept or reject based on the ratio between current policy and old policy;
Accepted tokens can be considered on-policy conditioned on the distribution, so no further importance correction is needed;
Rejected tokens continue with the normal correction process.

The article emphasizes that SIS can be directly plugged into existing policy gradient objectives such as GRPO, DAPO, and GSPO.

Engineering Implementation and Theoretical Properties

To reduce overhead, the authors use Top-K approximation to avoid computing the maximum ratio over the full vocabulary; since LLM distributions are concentrated, the extra error from this approximation can be bounded by the tail mass outside top-K. The article claims an additional overhead of about 1%.

Theoretically, SIS does not guarantee a completely unbiased gradient, but it makes tokens on the acceptance branch on-policy, and tightens the upper bound of bias towards the true gradient compared to vanilla token-level importance sampling.

Experiments and Observations

The authors present multiple experimental results:

Overall performance improvement across three backbones, three algorithms, and ten benchmarks;
Compared to GRPO, SIS (vanilla) already achieves significant improvement; adding DAPO's clip-higher yields the best results;
In stale rollout scenarios, SIS is more stable and has less degradation than baselines;
In MoE routing mismatch scenarios, SIS and R3 are complementary; their combination gives the best stability and accuracy.

Interpretability Observations

The authors also analyze differences between accepted and rejected tokens:

Accepted tokens are more related to mathematical reasoning (e.g., frac, sqrt);
Rejected tokens are more related to format characters and web artifacts (e.g., image, markdown tags).

Overall, the article shifts the off-policy processing approach from 'suppressing deviation' to 'turning some tokens back to on-policy', emphasizing simplicity, generality, and composability.

Original post →

SIS: Turning Off-Policy Tokens Back to On-Policy

More from Research