PR-OPD aligns hidden states for agentic RL, up to +14.0 WebShop accuracy over GRPO
Muyang Li · hf · 2026-10-14
A University of Florida-led paper identifies why token-probability self-distillation underuses privileged skills in agent RL: a skill lifts WebShop success from 42.2% to 56.2% yet changes fewer than a quarter of sampled token probabilities ("Invisible Advantage"), while shifting hidden states of over 80% of response tokens ("Much to Align"). PR-OPD has the policy write a hindsight skill per trajectory and align its projected hidden states to its skill-privileged self as a stop-gradient teacher at every layer, needing no external skill library, separate teacher, or inference overhead. It beats GRPO by up to 4.7 points on ALFWorld and 14.0 on WebShop across two backbones. Code is open-sourced.
More from Research
- Google Fellow John Platt: AI rerouting cut flight contrails by 62% — ziv_ravid · 2026-10-14
- Bible verse continuation triggers malevolent outputs, more evidence of subliminal learning — sebkrier · 2026-10-14
- ReRoute: physics-informed fine-tuning cuts counterfactual error up to 89.6% — AnimaAnandkumar · 2026-10-14
- Spork post-training recipe lifts chess puzzle solve rate 100% over best GRPO setup — aidenybai · 2026-10-14
- Dyna-2 Deep Dive: World-Action Models Trained on 1M Hours of Human Video, 99.4% 24-Hour Autonomy — Latent Space · 2026-10-14
- Procedural Core: a 1M-param abstract model boosts 85M-param ViT ImageNet accuracy by 2.2 points — kastnerkyle · 2026-10-14