Arrive and Survive: Scaling Safe Goal-Conditioned Policy Learning from One-Bit Failure Signals
Guopeng Li, Yiyang Duan, Yiru Jiao, Chengcheng Xu
cs.LG, cs.RO
2026-08-27
CRL overweights short near-crash futures after failure. Safe-CRL restores survival mass via weighted InfoNCE and log Z, beating Scaling-CRL on time-at-goal in all 12 robot tasks.
Contrastive RL turns sparse goal-reaching into self-supervision. From a state-action anchor, later achieved goals become positives, an InfoNCE critic learns a reachability score, and the actor climbs that score. Scaling-CRL showed the same objective keeps improving as actor and critic get deeper.
Failure termination breaks the accounting. A fall, a collision, or a pit ends the episode, so post-failure states are no longer valid future goals. Standard CRL keeps only the pre-failure futures and renormalizes their remaining occupancy to one. Two actions can then look equally well supervised if their surviving snippets visit similar goals, even when one snippet is a long stable trajectory and the other dies right after arrival. Short traces get amplified. Long traces get shrunk. The missing piece is survival mass: the probability mass that failure removes from discounted future-goal occupancy.
That bias shows up as two failure modes. Early replay is full of short near-crash traces, so the policy can lock onto falling (catastrophic failure bootstrapping). Or the agent hits the goal and immediately overshoots, collides, or falls (unsustainable reaching).
Safe-CRL changes two terms and nothing else. No extra cost label, no change to the action space. The one-bit termination flag is the only safety signal.
A 4-layer encoder estimates Z. Horizons H are drawn from the discounted geometric used by CRL; the label is whether failure occurs after H. Time-limit truncation without an observed failure counts as survival. Actor updates freeze encoder parameters but still route gradients through the action input. Everything else, including Scaling-CRL's log-sum-exp score regularizer, stays put. The coefficient on log Z is fixed at the theoretically required 1.
Twelve failure-prone Brax navigation and locomotion tasks, same setup as Scaling-CRL. Actor and critic are 64 layers deep except Point Goal and Car Goal (8 layers). γ=0.99, 1000-step time limit, five seeds.
| Task | Scaling-CRL time at goal | Safe-CRL |
| Ant Goal | 276.8 steps | 684.0 |
| Humanoid Goal | 22.2 | 266.4 |
| Humanoid Big Pitfall | 8.2 | 474.1 |
Mean time at goal and mean survival time rise on all 12 tasks. Goal coverage rises on seven and falls on five, but time at goal still jumps on those five, so the gains are not from skipping hard goals. On Ant Goal-like tasks, coverage barely moves while dwell and survival move a lot: the baseline reaches, then dies. On Humanoid tasks, Scaling-CRL survival stays low throughout training, closer to failure bootstrapping.
Depth still helps. 8-layer Safe-CRL already matches or beats 64-layer Scaling-CRL. Humanoid U-Maze keeps improving out to 256 layers. The 4-layer Z-encoder adds about 4.2% parameters and 2.8% wall-clock (9.07 to 9.32 hours per 100M steps). Ablations on shallower nets isolate the actor term: on 16-layer Humanoid Goal, time at goal goes 13.8 (baseline) / 28.3 (MW-InfoNCE only) / 288.7 (log Z only) / 297.0 (both). When goals respawn, the survival edge is larger than in a fixed-goal variant (Point Goal +13.8% vs +5.0%; Car Goal +18.3% vs +2.2%).
Plenty of goal-conditioned robot tasks already terminate on failure. That bit is free. Safe-CRL is the smallest patch that lets Scaling-CRL keep its depth scaling without a CMDP cost or a special failure token in the contrastive vocabulary. An 8-layer corrected policy beating a 64-layer uncorrected one is the practical takeaway.
It is an incremental fix aimed at a derived bias, not a new algorithm family. Rollouts still show S-shaped replans around moving gremlins, backward recovery, and slow detours around pits, all from the termination bit.
The authors list three. The method is for failure-terminated goal-conditioned control, not ongoing safety costs or constrained RL. Time-limit truncation is right-censoring; horizons without an observed failure are treated as survived, without the SVL censoring correction. Exploration is unchanged, so goal-space coverage stays uneven: left-back versus right-back success on a humanoid pitfall task is visibly asymmetric.
Ablations cover four tasks at reduced depth, so the huge Humanoid gaps should not be mentally copied onto every environment. β=1 is theory, not a tuned safety knob; larger β raises survival further and the effect on reaching depends on the task. Replacing the Monte Carlo Z label with TD learning hurts Car Goal and Humanoid Goal, which is why the default stays one-sample BCE.