RePPO trains a population of PPO policies, each solving the task its own way
TZahavy · x · 2026-11-17
New research work RePPO trains a population of PPO policies that all solve the task well, each in its own distinct way, probing the alternative solutions an RL agent could have found.
- Mechanism: each policy's reward combines the task return with a repulsion term in behavior space (r + pₖᵀm), pushing policies apart so they settle in different high-return regions (illustrated in the paper figure).
- Evaluated from locomotion to LLM finetuning.
- Some population strategies reportedly beat standard PPO.
More from Research
- BitDecoding-FP4: native FP4 KV cache decoding via algorithm-system co-design — UoE-Mlsys · 2026-11-18
- Third-party test suggests Claude was trained on protein sequences at scale — AllThingsApx · 2026-11-18
- Attention steering: a standout HCI experiment that directly manipulates LLM attention — round · 2026-11-18
- HT-Policy: multimodality isn't why diffusion policies win at imitation learning — chris_j_paxton · 2026-11-18
- Hypothesis: Higher Adam β1 May Help More in Small-Batch, Low-LR Training Regimes — torchcompiled · 2026-11-18
- NeurIPS paper: Joint Energy Models show human vision aligns between generative and discriminative — ShahabBakht · 2026-11-18