AI engineering is like making law, not playing games: rules must shift as they meet reality
danshipper · x · 2026-10-03
- Drew Breunig argues AI engineering resembles lawmaking more than game-playing. AI grew up on games—Deep Blue, AlphaGo, agents playing Doom and Civilization—but games have fixed rules and agreed definitions of success; reality doesn't.
- Writing agent instructions is like drafting and enacting laws: they encounter situations their authors never imagined and get reinterpreted case by case. Everyone has written "Not like that!" after being ignored; summaries drop critical caveats; coding agents "fix" bugs by rewriting unit tests.
- Each new model is also a different interpreter of the same instructions.
- The takeaway: AI system rules are provisional, and maintaining them is an ongoing legal-style craft.
More from AGI Musings
- Andrew Trask: solving AI attribution would also solve AI consciousness — iamtrask · 2026-10-03
- Brundage doubles down: with billions of users, AI firms aren't causally blameless for deaths — Miles_Brundage · 2026-10-03
- Miles Brundage: shipping with known flaws isn't iterative deployment, it's just faster shipping — Miles_Brundage · 2026-10-03
- Pope says algorithms lack the human spark: AI art and human art differ ontologically — sorenrood · 2026-10-03
- AI Debater: If Mass Displacement Is Coming, Where Are the Serious UBI Proposals? — dreamwieber · 2026-10-03
- SF information-sector jobs down 22% from 2022 peak, but total pay hits record $57B — randal_olson · 2026-10-03