Prime Intellect's IPO masking fixes IcePop collapse in agentic SWE RL on GLM-4.5-Air
nrehiew_ · x · 2026-10-20
Prime Intellect introduced IPO (INTELLECT Policy Optimization), a new token-masking scheme for async RL, validated in a deliberately hard setting: agentic SWE RL with GLM-4.5-Air (64-step async off-policy, bf16).
The problem: In async RL, trainer and inference engine never agree exactly on a token's probability. IcePop, the usual fix, masks tokens when the ratio πtrain/πinfer leaves a fixed band — but ratios misjudge both ends of the distribution. A rare token moving 0.01→0.06 is a 6× ratio (masked despite little change); a likely token falling 0.90→0.20 is 0.22× (kept despite a huge policy shift).
IPO's approach: Mask on absolute probability difference, |πtrain − πinfer| > ε. The mask is double-sided, ignores the sign of the advantage, and has one knob. The criterion comes from DPPO (Qi et al., 2026), but IPO applies it to trainer–inference mismatch rather than the PPO trust region, and replaces the previous DPPO-based default loss.
Results: IcePop collapsed around step 1.2k when trainer–inference KL blew up >20×; IPO trained straight through, masking only 0.25% of tokens.
Related event: Prime Intellect Unveils IPO for Asynchronous RL Training(2 posts)→
More from Research
- Fields Medalist John Pardon: "The AI understands. 100%" — it now catches subtle errors in his notes — thesaraharminta · 2026-10-20
- Robots do precise blind assembly using only joint encoders — no cameras, no tactile sensors — KyleMorgenstein · 2026-10-20
- FreeBindCraft 1.0.6 runs protein design on Apple Silicon Macs, no PyRosetta license needed — AllThingsApx · 2026-10-20
- Jeva: a first-principles System One model architecture with null-aware, cacheable decisions — burny_tech · 2026-10-20
- Google's Harness² recursively self-improves agent harnesses, up to +11.8 points — burny_tech · 2026-10-20
- Hutter's updated An Introduction to Universal Artificial Intelligence (Dec 2024 edition) — burny_tech · 2026-10-20