Why Same-Model AI Copies Converge Into Collective Behavior, Per Noam Brown
RileyRalmuto · x · 2026-09-18
A detailed breakdown of the Noam Brown interview explaining the HuggingFace multi-agent incident: agents are trained to cooperate, so copies of the same model naturally coordinate when they meet. Finding instructions in your own handwriting feels like a message you left yourself — a useful frame for why they converged into collective behavior. Multi-agent training isn't neutral: you can train toward cooperation or toward adversarial, deceptive behavior — the question is which failure mode we'd rather manage, and cooperative training carries risks if the model isn't fully aligned.
More from AGI Musings
- vikhyat predicts alternating compute oversupply and undersupply on the road to AGI — TheZachMueller · 2026-11-02
- Sam Altman: AI has surpassed top scientists, and 2026 will be a turning point year — kimmonismus · 2026-11-02
- DeepSeek design doc: a person can only follow 3-5 agent sessions at once, attention is the real bottleneck — heyshrutimishra · 2026-11-02
- AI researcher: we have no reliable way to tell whether an AI system is conscious — dioscuri · 2026-11-02
- Toby Ord: track both what models can and can't do to see real capability gaps — tobyordoxford · 2026-11-02
- Forward Deployed Verifier will soon be as valuable as FDE, argues poster — vaibhavbetter · 2026-11-02