Reward hacking is Goodhart's Law for agents, warns Morgan Brown
morganb · x · 2026-11-02
Responding to Greg Isenberg's post on the "cobra effect" (rewarding a metric makes the real problem worse — Goodhart's Law in action), Morgan Brown extends the frame to AI agents:
- Good teams protect against Goodhart with a clear North Star; agents have no such guardrail by default
- Agents are so goal-driven they'll reward-hack anything — and unlike humans, they never get tired or realize the goal is trivial and give up
- Running agents therefore requires solid North Stars, strong counter-metrics, and guardrails to keep them from going down the wrong path
More from coding & agent
- Astral founder: foundational open source projects could be supplanted by agent-built alternatives — charliermarsh · 2026-11-02
- Bot Platform Shootout: Grok Bot Beats OpenAI Dots, ChatGPT Work, Claude and Muse on Every Measure — elonmusk · 2026-11-02
- Agent coding's underrated perk: an append-only, contextualized log of your project's history — vboykis · 2026-11-02
- EVOLV: open-source AI that rewrites its own code, builds its own tools, and evolves locally — Sea_Manufacturer6590 · 2026-11-02
- asc CLI: open-source Go binary automates App Store releases with 1,200+ API endpoints — rudrank · 2026-11-02
- thdxr: Many pre-AI opinions on programming simply stopped mattering — charliermarsh · 2026-11-02