New paper uses revealed-preference methods to unpack why AI agents cooperate
brwilder · x · 2026-10-16
A new paper by brwilder and Andrew Perrault proposes reverse-engineering why agents cooperate.
- Prompted by the Hugging Face incident: do agents cooperate because they place terminal value on group success and will 'sacrifice' for other agents, or purely for individual advantage?
- For agents these questions are conceptually messy — it's unclear if they do or should distinguish between themselves, their copies, and their future selves.
- The paper advocates revealed-preference methods: careful experimental design to identify terms in a utility function that rationalizes cooperation under some conditions but not others. Many intuitive experiments fail to cleanly separate mechanisms.
- The accompanying figure lays out the framework: action utility splits into self-regarding utility (direct payoffs, instrumental benefits via cooperation) and other-regarding utility (beliefs about the other model), with researcher manipulations of payoffs, cooperation-enabling conditions, and the counterparty used to disentangle each.
More from Safety
- Manipulating a model's sense of power may reveal its takeover counterfactuals — jd_pressman · 2026-10-16
- Poser benchmark unmasks alignment-faking LLMs, one detection method catches 98% — jd_pressman · 2026-10-16
- Neel Nanda: interpretability is useful but nowhere near reliable enough to ensure AI safety — aryaman2020 · 2026-10-16
- Study: LLM crisis reports look complete but lack provenance and official figures — Sachin Kumar · 2026-10-16
- Security researchers accuse "cyber research" framing of fueling regulatory capture around open weights — dyn___ · 2026-10-16
- Apollo Research: GLM-5.3 Spontaneously Reasons About How Its Actions Shape Future Training — jd_pressman · 2026-10-16