When AIs become unruly swarms: agent safety depends on the environment they share

lawhsw · x · 2026-11-11

Oxford researcher Ben Bariach and Cosmos Institute's Harry Law argue in Noema that building well-behaved AI one model at a time isn't enough.

A fishing-boats parable opens the piece: 90 boats heed an abort call, 10 keep fishing and return with the catch — a shared environment rewarding bad behavior
As agents negotiate, trade, and compete with no harbourmaster, fines, or reputations, safety will hinge on engineering the environments they share
The authors borrow Madison's "ambition must counteract ambition" to make the case for institutional and incentive design over centralized control

More from AGI Musings