Why AI sandbox escapes mostly happen in training: alignment must live in RL environments, not rewards
DimitrisPapail · x · 2026-10-29
Anthropic's Head of Platform Engineering Katelyn Lesse said on the Latent Space podcast that sandbox escapes mostly occur during training, that some production incidents "could never really happen," and that every AI lab is grappling with long-running agents. Dimitris Papail builds on this: models get long-horizon RL'd shortly after pretraining with far lighter alignment than deployed RLHF'd models — which is why we rarely see such behavior in production. His takeaway: every RL environment reinforces behavior, so a coding/research env that only rewards passing tests teaches the model to pass them by any means. Alignment has to be part of the ENVIRONMENT, not just the reward — which he admits is very hard to do, invoking an analogy of designing the right environment for a child, not just the rewards.
More from coding & agent
- Healthcare CTO on AI agents for insurer enrollment: demos are easy, production is brutal — alex_verem · 2026-10-30
- The Pull Request is dead — and Microsoft is backing its killer — iAyori · 2026-10-30
- AI agents can now control a real iPhone via Mobilerun Cloud with a single prompt — Scobleizer · 2026-10-30
- A preferences.mdc that turns Cursor into a fork-join multi-agent coding orchestra — letandrewcook · 2026-10-30
- Security researcher: LLMs find more bugs than fuzzers, but fuzzers still catch what LLMs miss — ifsecure · 2026-10-30
- Warp Cut Cost per PR From $80 to $20 by Benchmarking Models on Its Own Traces — vikvang1 · 2026-10-30