Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models
Kurtland Chua, Roberto Calandra, Rowan McAllister, Sergey Levine
cs.LG, cs.AI, cs.RO, stat.ML
2018-05-31
PETS models dynamics with probabilistic ensembles, propagates uncertainty via trajectory sampling, and plans with MPC, matching model-free PPO/SAC asymptotic performance on MuJoCo at 1/125 (PPO) and 1/8 (SAC) the samples on half-cheetah.
This 2018 paper asks one question: can model-based RL (MBRL), which is sample-efficient, finally match the asymptotic performance of the best model-free algorithms, especially with deep networks as the dynamics model? The root cause is that neural networks overfit on small data, so multi-step predictions blow up, and planning on a bad model learns badly. Efficient models like Gaussian processes (GPs) learn fast but cannot handle discontinuous, high-dimensional contact dynamics in robotics. The authors want deep-network capacity, robustness on small data, and model-free-level final performance.
PETS (Probabilistic Ensembles with Trajectory Sampling) feeds both kinds of uncertainty to the model. Aleatoric uncertainty (inherent data noise) is captured by a probabilistic network outputting a Gaussian; epistemic uncertainty (subjective uncertainty from limited data) by an ensemble of 5 bootstrapped models that disagree where data is sparse, with disagreement signaling what the model does not know.
Planning uses MPC: at each step the cross-entropy method (CEM) samples candidate action sequences, the model rolls the current state into trajectories of 20 particles, expected return is evaluated, only the first action executes, and the next step replans. How particles propagate is the key, called trajectory sampling: each particle binds to one bootstrap model and samples forward from that model's probabilistic prediction. TS-infinity keeps a particle on the same model across the whole trajectory, separating epistemic from aleatoric uncertainty; TS-1 resamples the model each step. The design welds honest uncertainty expression onto model-based planning.
On four MuJoCo continuous-control tasks (cartpole, pusher, reacher, half-cheetah), compared against the strongest model-free (PPO, SAC, DDPG) and model-based (Nagabandi's deterministic nets, several GPs) methods of the time.
| Task | Key result |
| half-cheetah | Matches PPO/SAC asymptotic performance at 1/8 the samples of SAC and 1/125 of PPO |
| All four | Reaches PPO's asymptotic performance in under 100 trials (100K steps) |
The model-choice ablation is telling: model-type ranking is PE above P above DE above D, with ensembles that capture both uncertainties strongest, single-uncertainty models next, and deterministic nets worst. The propagation technique (E, MM, DS, TS variants) matters far less; the ceiling is set by how the model represents uncertainty, not by propagation tricks. GP-MM only edges PETS on low-dimensional cartpole but scales cubically in time and quadratically in state dimension, so it cannot run on the higher-dimensional tasks.
This was the first time model-based deep RL genuinely matched model-free asymptotic performance on standard benchmarks, breaking the old view that model-based methods are efficient but ceiling-limited. For robotics and real-world systems, sample efficiency is the difference between sim-only and real hardware: 1/125 means a skill that once took millions of steps now takes tens of thousands. The lesson to explicitly model uncertainty became a main line in model-based RL and uncertainty quantification.
The cold-water reality: there is no policy network, so testing relies on online MPC, which is expensive. The authors tried to train a parameterized policy directly from this uncertainty-aware model, but gradients diverged through the model (chaotic gradients), and it failed, left as future work.
The authors note they did not use epistemic uncertainty for directed exploration (only passively, to avoid overfitting), and that the policy-learning path did not work.
Two points from today's vantage: validation is limited to MuJoCo continuous control at low-to-medium dimension, still far from image inputs, high dimension, and complex real-robot contact; and online MPC's compute cost is a hard constraint for real-time control (the paper flags 20Hz and above). Sample-efficient and fast per step are different things.