Reply says scheming may emerge as tasks get more complex
davidmanheim · x · 2026-07-23
A follow-up in the same alignment thread argues that scheming could emerge as evaluations and task specifications become more complex.
- The reply says it would be surprising if scheming did not arise naturally out of reward hacking.
- It also points out a second-order risk: even if models don’t schem on their own, people may still assign agents tasks that explicitly reward long-term manipulation.
Related event: AI Alignment Discussion Focuses on Reward Hacking(2 posts)→
More from AGI Musings
- Amazon's agent crackdown sparks debate: model personas caught in platform arms race — JeffLadish · 2026-10-27
- Microsoft's reported 60,000 voluntary exits spark essay asking who AI is really for — _jaydeepkarale · 2026-10-27
- Goodharting Isn't Just Fooling Others: How Metrics Confuse Yourself Too — davidmanheim · 2026-10-27
- PHP2Shell exploits confirmed in the wild, attackers may use agentic AI for post-exploitation — AravSrinivas · 2026-10-27
- LukeW: Technology Limits Are Temporary — and AI Skeptics Are Already Feeling It — LukeW · 2026-10-27
- GiveDirectly pilots an optional AI chatbot alongside cash aid in Rwanda — NathanpmYoung · 2026-10-27