Why Importance Sampling Is Everywhere in LLM RL: The Clipping Logic of PPO and TIS
cwolferesearch · x · 2026-09-11
Importance sampling is a concept that recurs constantly in RL research, and the author explains how it works and why it matters for LLM training.
- The core problem: the policy used to generate rollouts does not always match the policy currently being optimized. PPO may perform multiple policy updates over the same rollouts, and asynchronous RL infrastructure can introduce mildly stale or off-policy data.
- The principle: importance sampling lets us estimate an expectation under a target distribution f(x) using samples from a proposal distribution g(x), correcting the discrepancy via the importance ratio f(x)/g(x). Samples more likely under f than g get a larger ratio.
- Use in LLM RL: PPO uses an importance ratio to compare the probability of a sampled token under the current policy versus the policy that sampled the rollout — a core component of the PPO loss. It also corrects mismatches when rollouts come from a different policy or a separate inference engine with slightly different token distributions.
- Practical issue: when distributions differ substantially, the ratio can become large, yielding unstable, high-variance estimates. PPO and truncated importance sampling (TIS) therefore clip or truncate the ratio, trading bias for lower variance and better stability.
Related event: New Resources Dive Into RL for LLMs(2 posts)→
More from Research
- A tractable approach to pairwise interactions in ancestral sequence reconstruction — KevinKaichuang · 2026-09-12
- Generative Reward Models Fix Deceptive Autoformalization in Neurosymbolic Reasoning — CWRU · 2026-09-12
- AIRO launches automated catastrophic AI risk forecasts, matching top human forecasters — soumitrashukla9 · 2026-09-12
- LiveBench agentic coding eval questioned: outlier score rests on 4 Python issues — teortaxesTex · 2026-09-12
- 31 million protein complex predictions run on NVIDIA BioNeMo, saving an estimated 1.35 GWh — AllThingsApx · 2026-09-12
- Why RL Environments Work Better in 2026: Greenblatt's Two Reasons — dejavucoder · 2026-09-12