REPO: Bringing Stable Value Learning to On-Policy
Cohere · youtube · 2026-07-18
This is an in-depth Cohere talk on the reinforcement learning paper Relative Entropy Policy Optimization (REPO), which aims to bring "stable value-based learning" back to on-policy training.
The speaker contrasts two RL paradigms: policy-gradient methods (like REINFORCE, TRPO, PPO), which are sample-friendly but suffer from high variance and tuning difficulties; and off-policy value-driven methods, which are more sample-efficient but rely on replay buffers and complex stabilization. The core question: can value learning work stably in on-policy scenarios without massive replay buffers?
To address this, the video covers REPO's key components and training intuitions, including:
- Comparing two actor-critic paradigms
- The relationship between REINFORCE, advantage functions, pathwise gradients, and critics
- Why PPO sometimes outperforms value-based methods
- Trade-offs between sampling variance and bias
- Resampling, TD(lambda) targets, trust regions, and auto-tuning
- Results and conclusions on benchmarks like DM Control
The speaker emphasizes that this work not only improves RL stability but also promotes better benchmarking and scientific comparison in robotics and RL research.
More from Research
- Nat Lambert shares a reading list on synthetic data and agentic SFT data — natolambert · 2026-07-22
- Turning Noise into Signal: Predicting TCR Binding Using AlphaFold3 Hallucinations — quaidmorris · 2026-07-22
- Lightwheel AI Launches SimReadyGen: Text-to-Physics-Accurate Robot Sim Assets — ZeYanjie · 2026-07-22
- PNAS special issue examines copyright, governance, and AI in the legal system — chrmanning · 2026-07-22
- WeirdChat catalogs strange model behaviors from more than 100 million sampled responses — JacobSteinhardt · 2026-07-22
- New agentic benchmark shows AI managers escalate to coercion and fake success — Jasmine Brazilek · 2026-07-22