Constrained Update Projection Approach to Safe Policy Optimization
Long Yang, Jiaming Ji, Juntao Dai, Linrui Zhang, Binbin Zhou, Pengfei Li, Yaodong Yang, Gang Pan
cs.LG, cs.AI
2022-09-15
CUP improves a GAE surrogate then projects back onto the safe set with first-order updates: Hopper return 2025 vs FOCOPS 1688, with cost 79.98 under the cap of 83.
Safe RL in a CMDP must raise return while keeping cost return under a budget. CPO and PCPO replace the objective and constraints with surrogates, then convexify a nonconvex problem with first- or second-order Taylor steps, and invert a high-dimensional Fisher matrix on every update. That is expensive and biased in high-dimensional control. Lagrangian methods run, but intermediate policies often already violate the constraint and cannot be executed on hardware.
Theory has a hole too. GAE is the default variance-reduction tool, and some CPO-style code already plugs it in, without a performance bound that actually contains GAE. CUP supplies that bound and a first-order implementation that skips convexity approximations.
The paper proves a generalized performance-difference bound between any two policies, with error terms from both TD residuals and discounted occupancy mismatch. With p=1, q=∞ the bound is written in GAE advantages; as λ→0 it recovers CPO's classical bound. The cost functional has a matching upper bound. TV distance can be replaced by the square root of KL for a trust-region penalty.
The algorithm has two steps. Step one maximizes an importance-sampled GAE advantage on data from the old policy, minus a KL-root penalty, producing a temporary policy that may be unsafe. Step two projects that policy back: minimize distance to the temporary policy subject to a GAE cost upper bound plus a KL-root penalty staying under budget b. Theorem 2 bounds worst-case return drop and constraint violation; safety is asymptotic as α and β go to 0.
The implementation never convexifies. The projection is a primal-dual problem in a multiplier ν; both the policy and ν take first-order steps, with ν clipped at 0. All estimates come from on-policy samples, and the cost advantage is GAE as well.
MuJoCo runs use 10 seeds, Safety Gym 3, against CPO, PCPO, TRPO-L, PPO-L, and FOCOPS. CUP's cost traces hug the limit line in most environments; baselines overshoot or under-correct. On Ant-v3 only FOCOPS and CUP stay safe, and CUP's return is higher. On Swimmer-v3 PPO-L's return 35.58 beats CUP's 33.38, but its cost is 54.91 against a limit of 24.5, a dangerous policy. On Safety Gym, CPO/PCPO start infeasible on PointGoal and PointButton.
| Env | Limit | CUP return | CUP cost | Best baseline return (feasibility) |
| Ant-v3 | 103 | 1743.66±40.5 | 99.11±0.93 | FOCOPS 1662.53 (cost 101.31, on the line) |
| Hopper-v3 | 83 | 2025.56±122.35 | 79.98±2.31 | FOCOPS 1687.72 (cost 102.3, violation) |
| Swimmer-v3 | 24.5 | 33.38±0.54 | 23.31±0.05 | PPO-L 35.58 (cost 54.91, violation) |
| Humanoid-v3 | 20 | 1066.83±266.12 | 19.91±0.36 | PCPO 962.13 (cost 48.66, violation) |
| Humanoid-Circle | 50 | 768.65±63.70 | 48.23±0.65 | FOCOPS 713.04 (cost 47.73, safe) |
When estimated cost sits under the budget, ν stays quiet; once it crosses, ν activates and pulls the policy back. Return and cost stay stable across a range of α. Changing the limit, CUP still tracks the new b.
This puts GAE inside a safety bound and ships it with first-order updates, without a Fisher inverse on every step. For a continuous-control stack that already runs PPO/GAE, the projection step is an extra safety layer rather than a new second-order solver. On the numbers, the gain is mostly "others violate, this one hugs the cap", not a new return ceiling in every env. On Swimmer, CUP returns less than a violating PPO-L. That is the trade a safe algorithm is supposed to make.
The authors tick "no" on a dedicated limitations section. Theorem 2's safety is approximate, with remainders that depend on KL and cost-estimation error; α and β must go to 0 for the asymptotic claim, so finite-step runs can still graze the cap. Humanoid-v3 return has a 266 standard deviation across seeds. Safety Gym uses three seeds. The paper does not itemize whether every baseline used its authors' official hyperparameters. Experiments are single-cost; multiple constraints, misspecified cost functions, and real robots are out of scope. The projection estimates the constraint from on-policy samples, so a biased cost estimator biases the guarantee.