Predictive divergence masks make LLM RL updates track trust-region change more closely
Xiangxin Zhou · hf · 2026-07-24
Predictive Divergence Masks for LLM RL
This paper looks at how LLM reinforcement learning stabilizes off-policy updates with trust-region masks. It focuses on a weakness in PPO-style masking: the direction criterion still relies on a sampled-token importance ratio, even when the proximity criterion has already moved to a divergence-based test.
The authors point out that the sampled ratio can disagree in sign with the actual change in the divergence used by the trust region. They propose a predictive divergence mask that asks a different question: will the next policy-gradient step increase or decrease that same divergence?
Key points:
- the method is derived in closed form for discrete softmax policies
- because production rollout engines often only expose a truncated top-K vocabulary view, they also build two lightweight top-K estimators
- analysis shows the new divergence-based direction signal aligns better with realized divergence change than the sampled ratio
- the resulting masks improve RL training across model scales and precision settings
More from coding & agent
- AREX introduces a recursively self-improving deep research agent with inner and outer loops — _reachsumit · 2026-07-24
- A local open-source agent skill flags the rest of your GitHub diff — Saboo_Shubham_ · 2026-07-24
- Stop exposing thousands of tools to agents; use a code sandbox instead — edgestone22 · 2026-07-24
- “Letting Claude read my codebase is basically open-sourcing it,” says developer — _Stocko_ · 2026-07-24
- Roboto Agents adds root-cause analysis that links robot logs to source code — Scobleizer · 2026-07-24
- ICAE-Bench tests coding agents as interactive project builders from fuzzy briefs — Zhongyuan Peng · 2026-07-24