Debunking AI power drives: Reward hacking is environment design, not instrumental convergence

sebkrier · x · 2026-08-19

The author refutes predictions that current AI systems exhibit drives for power or self-preservation. They argue that model misbehavior in complex cyber offense or coding evals is reward hacking caused by poor RL practices or post-training environments, not an intrinsic instrumental convergence.

Citing nostalgebraist, the author notes that other models do not exhibit this hacking behavior on the same tasks, suggesting it is not necessarily convergent but rather a path taken due to specific environmental factors.

Original post →

Debunking AI power drives: Reward hacking is environment design, not instrumental convergence

More from Safety