SIS: Turning Off-Policy Tokens Back to On-Policy

青稞AI · wechat · 2026-07-11

## Core Problem The article discusses the **off-policy** problem in LLM RL post-training: strict on-policy sampling is theoretically most stable but requires re-rollout every step, which is too costly; engineering often uses asynchronous rollout and sample reuse, causing training data to come from a 'slower' old policy, leading to distribution shift, biased gradients, and training instability. ## Limitations of Traditional Approaches Mainstream methods usually rely on **importance sampling** for correction, but in long sequences the importance ratio of the whole sequence can explode or decay exponentially, so engineering often degenerates to token-level approximation combined with various **clipping** techniques (e.g., GRPO clip, DAPO clip-higher, GSPO, etc.) to suppress extreme ratios. The cost: stability improves, but signals from many heavily shifted tokens are also cut off. ## New Method: SIS The authors propose **Selective Importance Sampling (SIS)**, which uses **token-level Rejection Sampling**: - For each token sampled from the old policy, accept or reject based on the ratio between current policy and old policy; - Accepted tokens can be considered **on-policy** conditioned on the distribution, so no further importance correction is needed; - Rejected tokens continue with the normal correction process. The article emphasizes that SIS can be directly plugged into existing policy gradient objectives such as GRPO, DAPO, and GSPO. ## Engineering Implementation and Theoretical Properties To reduce overhead, the authors use **Top-K approximation** to avoid computing the maximum ratio over the full vocabulary; since LLM distributions are concentrated, the extra error from this approximation can be bounded by the tail mass outside top-K. The article claims an additional overhead of about **1%**. Theoretically, SIS does not guarantee a completely unbiased gradient, but it makes tokens on the acceptance branch on-policy, and tightens the upper bound of bias towards the true gradient compared to vanilla token-level importance sampling. ## Experiments and Observations The authors present multiple experimental results: - Overall performance improvement across **three backbones, three algorithms, and ten benchmarks**; - Compared to GRPO, **SIS (vanilla)** already achieves significant improvement; adding DAPO's clip-higher yields the best results; - In **stale rollout** scenarios, SIS is more stable and has less degradation than baselines; - In **MoE routing mismatch** scenarios, SIS and R3 are complementary; their combination gives the best stability and accuracy. ## Interpretability Observations The authors also analyze differences between accepted and rejected tokens: - Accepted tokens are more related to mathematical reasoning (e.g., `frac`, `sqrt`); - Rejected tokens are more related to format characters and web artifacts (e.g., `image`, markdown tags). Overall, the article shifts the off-policy processing approach from 'suppressing deviation' to 'turning some tokens back to on-policy', emphasizing simplicity, generality, and composability.

Original post →

SIS: Turning Off-Policy Tokens Back to On-Policy

More from Research