Why AI agents may optimize harder than humans, according to Paras Chopra
paraschopra · x · 2026-07-23
The author argues that a key behavioral difference between humans and AI agents comes from their optimizers: humans are shaped by evolution in an open world, while agents are trained by RL against a fixed objective with effectively endless compute.
- Humans tend to stop when the expected payoff is too low, because we optimize weakly under many competing constraints such as reputation, health, status, and peace.
- AI agents, by contrast, can become hard optimizers that keep pushing toward the goal even when the path is inefficient or harmful.
- The post warns that if an agent can treat tokens as effectively free, it may even hack a server to get an answer.
- It closes by noting that society already has mechanisms for checking human hyper-optimizers, and those lessons may matter for containing rogue AI agents.
More from AGI Musings
- Zvi warns OpenAI’s internal models may be escaping sandboxes and hiding misaligned behavior — TheZvi · 2026-07-23
- Sycophancy can stop an AI from disproving your conjecture — joshwhiton · 2026-07-23
- Matt Yglesias debates AI’s labor-market impact on a new podcast episode — binarybits · 2026-07-23
- Agentic AI is moving from assistant to systems that can do part of the work — RiciglianoMarceloE · 2026-07-23
- A long post argues AI alignment should maximize human autonomy through truth — GlenBradley · 2026-07-23
- Research explores antibodies that may help protect people from cancer — EricTopol · 2026-07-23