48-page deep dive: why modern LLM RL is secretly off-policy and 14 ways to fix it
cwolferesearch · x · 2026-10-31
Chris Liu publishes a 48-minute-read survey, "Off-Policy Drift in LLM RL," arguing modern LLM RL is secretly off-policy even in frameworks advertised as on-policy. Drift sources include separate inference/training engines at different precisions, different attention kernels, tensor-parallel configs, MoE routing flips, async pipelines consuming stale rollouts, and multi-epoch training. The post catalogs 14 advantage estimators and correction methods (R3, IcePop, QUATRO, MinPRO, SAPO, GSPO, OAPL, KL-cov, R²VPO, DISPO), verifies production recipes against the verl repo including DeepSeek-V3.2 and Ring-1T, and closes with five falsifiable conjectures. Lambert and cwolferesearch are reading and recommending it.
More from coding & agent
- Chrome sandbox escape still lurks in cached Playwright Chromium inheriting your IDE's macOS permissions — flngr · 2026-10-31
- How to Give Your AI Dot Real Responsibilities: Beyond Reminders to Ongoing Delegation — nickbaumann_ · 2026-10-31
- Socket: Malicious Claude Code 2.1.313 Extension Removed From Cursor Marketplace, Early Reports Overblown — zeeg · 2026-10-31
- Dev laments spending 75% of his time waiting for coding agents to finish — RileyRalmuto · 2026-10-31
- Dev says Opus 5.5 leads on productivity, keeps GPT Pro subscription anyway — kevinkern · 2026-10-31
- GitHub demos Chopin, an experiment in multiplayer planning for the agentic era — 0xkarasy · 2026-10-31