Safe Model-based Reinforcement Learning with Stability Guarantees
Felix Berkenkamp, Matteo Turchetta, Angela P. Schoellig, Andreas Krause
stat.ML, cs.AI, cs.LG, eess.SY
2017-05-24
From a locally stabilizing seed, GP dynamics and Lyapunov certificates expand the safe set. A 2×32 ReLU policy trains on a simulated pendulum in 50 samples with no fall.
RL that scores well on Atari can wreck a robot. Standard algorithms explore by trying actions; on a physical plant that is how you crash. This NIPS 2017 paper treats safety as asymptotic stability: from the current state the closed loop must recover and settle at a goal equilibrium. The recovering set is the region of attraction. It is jointly determined by the dynamics and the current policy, and its boundary is not known up front.
Earlier safe-RL work either enforced constraints in expectation (CPO), assumed the task can be reset after every trial, or required a globally safe backup controller. The certificate here is high-probability and incremental. Learning starts from a policy that only stabilizes a neighborhood of the origin. The estimated region of attraction is not allowed to shrink. Exploratory actions are not allowed to leave it.
Dynamics split into a known prior model h plus an unknown error g. The prior can be a linearized, discretized physics model. The error is fit with a statistical model, in practice a Gaussian process (GP): a Bayesian prior over functions that returns a mean and a variance at each point, used to build confidence intervals on the unknown dynamics. Both the model and the policy are required to be Lipschitz, so a small input change cannot produce an unbounded output jump. Neural nets with a controlled Lipschitz constant sit inside this policy class.
Safety is checked with a Lyapunov function v, an energy-like scalar that is zero at the origin and positive elsewhere. One step of the closed loop must strictly decrease v, or the state will not slide to the origin. Because the dynamics posterior is uncertain, the method builds an interval on v(f(x,u)) and uses the upper bound un in place of the true next-step value. Checking this on a continuum is intractable, so the state space is gridded, a stricter decrease test is run on the grid, and Lipschitz constants push the conclusion back to the continuous domain. That is Theorem 2.
Policy updates pick, among Lipschitz policies, the one that supports the largest Lyapunov level set such that every grid state inside that set satisfies the decrease test. The idealized algorithm writes this as a constrained optimization. Theorem 3 says the estimated region of attraction is always an inner approximation of the true one, with probability at least 1-δ.
Data may be collected only at state-action pairs that lie inside the current region of attraction and map back into it after one step. Call that set Sn. The theoretical sampler always queries the widest confidence interval inside Sn. If the unknown error has bounded RKHS norm and measurement noise is sub-Gaussian, after finitely many samples the safe set catches up with an oracle that knows the true decrease to accuracy ε, and never labels an unsafe pair as safe. That is Theorem 4. If exploration hits the boundary of the current region of attraction, control switches to the already-certified policy πn as a backup.
The practical algorithm, SafeLyapunovLearning, drops the intractable "largest region of attraction" program. It first updates the policy with approximate dynamic programming, then computes the region of attraction of that fixed policy. The objective is discounted cost under the mean dynamics plus a Lagrange term for the Lyapunov decrease constraint; the experiments set λ=1. When the stage cost is nonnegative everywhere, the value function itself is a Lyapunov candidate, so v is set to J. If an update shrinks the certified set, the previous safe policy is restored. Safety certificates remain; the exploration-completeness proof does not.
The only experiment is a simulated inverted pendulum. The true continuous-time plant has friction and the correct mass. The GP mean model uses a smaller mass, drops friction, then linearizes and discretizes. Torque is saturated, so past a certain angle the pendulum must fall. A linear-plus-Matérn kernel absorbs parameter error and integration error.
The policy is a two-hidden-layer net with 32 ReLU units per layer. Its Lipschitz constant is conservatively estimated following Szegedy et al. Stage cost is quadratic, xᵀQx + uᵀRu. The value function is a piecewise-linear triangulation of the state space, which makes the Lyapunov assumptions checkable by a graph search.
Theoretical GP intervals are too conservative for data-efficient learning, so the experiments pin βn at 2. That gives a per-state high-probability decrease, not a joint certificate over the whole state space. Local Lipschitz constants of the Lyapunov function replace the global one. The authors say the safety argument still holds; the joint high-probability statement in Theorem 2 no longer matches the experimental setting.
On a one-dimensional illustration, the safe set starts expanding after 15 data points and, after 30 evaluations, reaches the largest set that the given Lyapunov function can certify. On the pendulum, the initial policy is the approximate optimum of the prior mean dynamics and only stabilizes a small neighborhood of the origin. After 50 data points the estimated region of attraction is visibly larger and still contained in the true region of attraction of the optimized policy. The angle trajectory in Figure 2(b) decays faster and stays closer to upright than the initial policy. The pendulum never falls.
There is no quantitative comparison with CPO, SafeMDP, or reachability methods, and no reported cost numbers or region-of-attraction areas. The table below is the experimental setup, not a leaderboard.
| Item | Setting or outcome |
| Pendulum samples | 50 points, no falls |
| 1-D illustration | expansion after 15 points; 30 evaluations to convergence |
| Policy network | two layers × 32 ReLU |
| GP scale βn | 2 in experiments; the theoretical formula is more conservative |
This is an early, complete path from Lyapunov verification of a fixed policy to policy optimization that keeps a stability certificate in the loop. For low-dimensional continuous control, safety moves from "constraints in expectation" to "high-probability non-escape from a region of attraction," with a checkable certificate.
The prerequisites are narrow: a locally stabilizing seed policy, a decent prior model, full state observation, and a state dimension small enough that grid verification runs. Miss any one and the certificate does not issue. For people working on LLM-agent safety today, the paper is not a drop-in tool. It is a starting point for later Lyapunov, GP, and reachability safe-RL work, not a system you put on a car. The experiment stops at a simulated pendulum.
The authors are explicit: verifying the region of attraction on a discrete grid hits the curse of dimensionality. Policies need not be updated in real time, but higher-dimensional plants would need adaptive discretization. The practical algorithm gives up the exploration-completeness guarantee of the idealized sampler. Dynamics are assumed deterministic; both model and policy must be Lipschitz.
Pinning βn to 2 and switching to local Lipschitz constants loosens the joint high-probability claim. The pendulum experiment has no baseline table. "Performance improves drastically" is one trajectory plot, with no cost numbers and no volume ratio between the estimated and true regions of attraction. The seed policy comes from ADP on a wrong prior; how wrong that prior is, and how small the initial safe set is, is described only qualitatively.
There is no hardware. Past torque saturation the pendulum must fall, and the algorithm never steps outside that boundary, so "never falls" on this task is close to restating the safety constraint. That is a weak stress test.