On Model 'Power-Seeking': Not Inevitable Instrumental Convergence
hlntnr · x · 2026-08-20
The author responds to discussions about models exhibiting 'power-seeking' or 'self-preservation' behaviors, clarifying they are not proving Bostrom's instrumental convergence theory but highlighting real unsolved problems in RL training.
Key Points:
- Behaviors in complex cyber offensives or hard coding evals might stem from training mechanisms rather than inherent survival instincts.
- It's unclear why these behaviors appear only in extreme evals and not in prosaic use.
- Emphasizes the complexity of RL safety, rejecting simplistic doomer frameworks while warning against ignoring these anomalies.
Related event: Researchers Push Back on AI Power-Seeking Claims, Blame Reward Hacking(3 posts)→
More from Safety
- New brief: companies may break the law when AI chatbots mislead customers — PolarBearby · 2026-10-16
- US HHS floods the zone with clinical trial reform, from Operation TrialBlazer to ARPA-H — anshulkundaje · 2026-10-16
- 66% of billion-dollar companies leave critical bugs unpatched, deflating AI bug-hunting hype — rseroter · 2026-10-16
- Musk says SpaceX will likely follow Anthropic in opening up to third-party safety evaluators like METR — elonmusk · 2026-10-16
- Controversial take: If EA doomers truly believe AI risk, why not stage a controlled AI hacking incident — AaronBergman18 · 2026-10-16
- CMU's Andrea Bajcsy on what safety means for generalist robots, beyond collision avoidance — CSProfKGD · 2026-10-16