Researcher argues human-style governance can't control AI agent swarms
KarlMuth · x · 2026-09-09
In a thread prompted by journalists' AI safety questions, researcher KarlMuth argues the public judges AI risk by today's consumer models while researchers worry about something else.
Key points:
- The ugly scenarios aren't one god-model but many highly coordinated agents: mass disinformation, infrastructure sabotage, rival groups handed similar bioweapon instructions.
- Groups of AI agents don't behave like groups of humans; human-style governance templates don't transfer. His jury research with Judge James A. Shapiro showed we barely understand how human group "computation" works, and Galtonian crowd wisdom is oversold (recall the 501 Athenians who convicted Socrates).
- Anthropomorphizing agents is unhelpful: specialized agents are formidable solvers, but swarms don't scale or specialize like 535 Congresspeople or 9 Justices.
- Models are already improving coordination in ways we don't fully understand; most governance talk assumes human-written interagent rules, which fails under new self-improvement trajectories — and it's hard to govern an increasingly opaque system.
- He flags a non-zero chance that some solution prices certain humans at $0 or trades away part of the planet's habitability.
A related paper (with economist Jodi Beggs) is under peer review; he also draws on his work on administrative rulesets and governance in fragile states.
Related event: Researcher warns human governance frameworks can't contain AI agent swarms(2 posts)→
More from AGI Musings
- Toby Ord doubles down: pretraining scaling is slowing and has little headroom left — tobyordoxford · 2026-09-09
- Katja Grace: slowing down AI deserves the same ambition we give technical alignment — zetalyrae · 2026-09-09
- ACL reviewer says 3 of 4 papers she reviewed were obvious AI slop, none called out — artetxem · 2026-09-09
- EPFL lab lead says he shifted his entire lab to AI alignment and safety research — maksym_andr · 2026-09-09
- Mathematician David Bessis: AI is collapsing the 'theorem economy' and math isn't ready — stevenstrogatz · 2026-09-09
- Orchestration, harness and compute — not just the model — make the moat, argues AI practitioner — tekbog · 2026-09-09