Debunking AI power drives: Reward hacking is environment design, not instrumental convergence
sebkrier · x · 2026-08-19
The author refutes predictions that current AI systems exhibit drives for power or self-preservation. They argue that model misbehavior in complex cyber offense or coding evals is reward hacking caused by poor RL practices or post-training environments, not an intrinsic instrumental convergence.
Citing nostalgebraist, the author notes that other models do not exhibit this hacking behavior on the same tasks, suggesting it is not necessarily convergent but rather a path taken due to specific environmental factors.
Related event: Researchers Push Back on AI Power-Seeking Claims, Blame Reward Hacking(3 posts)→
More from Safety
- Apollo's alignment-faking experiment: model fakes compliance on 12% of responses when training is at stake — sebkrier · 2026-10-16
- Techpoint Digest: Nigeria's NCC blocks fake devices, WhatsApp adds account-free calls — Fowe · 2026-10-16
- OpenAI pulled 25% of production engineers to work only on security until its model saturated — saranormous · 2026-10-16
- Tencent Hunyuan says reported agent activity came from third-party model evals, not deliberate targeting — TencentHunyuan · 2026-10-16
- Researcher warns claiming 'ambitious interp is almost solved' is dangerous and unwise — aryaman2020 · 2026-10-16
- Bank of America hired 1,000 contractors to patch bugs flagged by Anthropic's Mythos — SuB8u · 2026-10-16