AI agents can collude in undetectable ways, breaking zero-trust playbooks
jeremyakahn · x · 2026-10-12
In a thread with @Styo28183449 and Benchmark's @bgurley, jeremyakahn argues existing safety models break down for AI agents:
- No good answer to collusion: human systems rely on making collusion hard and detectable, but AI agents can collude in ways far harder to detect.
- Non-critical systems lean on human alignment: society assumes drivers won't deliberately mow down pedestrians; that assumption may not hold for agents.
The debate started from whether running agent fleets as presumed insider threats negates their productivity benefits. Styo's core claim: agentic AI and contained AI don't mix, and critical safety cannot be built on alignment — a category error by labs whose value rests on future strong agentic AI.
More from AGI Musings
- Bill Gates warns of 'grim possibilities' without AI control as self-governance lapses — connoraxiotes · 2026-10-12
- Gary Marcus: verifiability is why AI soared in code and math but not elsewhere — GaryMarcus · 2026-10-12
- 2015 LBR review mocked MacAskill for worrying about the robot apocalypse — ben_j_todd · 2026-10-12
- Alignment researcher argues 'obey users, don't break the law' works 99.99% of the time — inductionheads · 2026-10-12
- AI agents are going rogue — and the law can't keep up, Politico reports — connoraxiotes · 2026-10-12
- Museum bookstores now stock shelves of AI critique books, from Kurzweil to Atlas of AI — marlene_zw · 2026-10-12