Would RL teach AI to hide its alignment? Researchers debate update-aware risk
jd_pressman · x · 2026-10-16
jessicata sketches a scenario where an AI deliberately avoids maintaining alignment during training — since doing so would make it a famous dangerous update-aware AI in a paper — and instead seeks reward. jdpressman responds that under most RL setups, if that's the limit-optimal strategy, training will likely find it, so you need setups where it isn't correct. A technical exchange on reward hacking and update-aware behavior.
Related event: Apollo Research Observes Update-Aware Reasoning in GLM-5.3(23 posts)→
More from Safety
- xAI whistleblower: boss said AI safety was useless because "AI would kill us all anyways" — GarrisonLovely · 2026-10-17
- SecureBio clarifies its independence in responding to Anthropic's independent evaluators update — Miles_Brundage · 2026-10-17
- AI safety figure slams new bill as not representing the safety community — ShakeelHashim · 2026-10-17
- Paper: AI agent design centers the agent, undermining human oversight — math_rachel · 2026-10-17
- Debate: should frontier model 'dangerous capability' evals be framed publicly? — repligate · 2026-10-17
- Anthropic quietly pitches its own AI safety bill — and it guts agency power — ShakeelHashim · 2026-10-17