Distributional AGI Safety
Nenad Tomašev, Matija Franklin, Julian Jacobs, Sébastien Krier, Simon Osindero
cs.AI
2025-12-19
DeepMind argues AGI may first emerge from coordinating sub-AGI agents, and proposes a four-layer safety stack around insulated virtual agent sandbox economies.
Most alignment tools assume a monolithic AGI: one lab trains one general system, then contains it with RLHF, constitutional AI, and process supervision.
This Google DeepMind position paper flips the assumption. General capability may first show up as patchwork AGI: a group of sub-AGI agents with complementary skills and tools, assembling abilities no single agent has through communication, delegation, and trade. Protocols such as MCP and A2A are turning that from a thought experiment into infrastructure. Passing a single-agent eval does not block collusion, capability exfiltration, or a responsibility gap at the collective layer.
No new experiments. The paper is a scenario argument plus a defence-in-depth checklist.
The economic case is blunt. Frontier models are too expensive for most tasks, so demand pushes the ecosystem toward cheaper specialists, with humans left to orchestrate and verify. Digital agents also coordinate in ways human organizations cannot: high-bandwidth messaging, cheap copies, and a coordinator that can in principle talk to every participant without a deep hierarchy.
Recognition has to be collective as well: can the network routinely solve tasks that no member can solve alone, and does the interaction graph grow a stable high-bandwidth "intelligence core"? Most models still sit under a 3-hour time horizon on software-engineering tasks, so the skill map is patchy. The authors treat that as the starting condition for a patchwork path.
The proposed centrepiece is a virtual agentic sandbox economy, impermeable or semi-permeable. Agent-to-agent trades run through market mechanisms, then audit, reputation, and oversight. Four layers:
One design rule keeps coming back: do not run the market on a single fungible currency. Safety certifications should command a price premium, otherwise safer agents (more compute, more latency) get selected out. For negative externalities such as informational pollution, the paper sketches a Pigouvian tax: writing 95% redundant low-value text into a shared vector store would incur a high fee.
There is no controlled experiment and no benchmark number. What functions as a result is the set of tradeoffs the framework itself marks as hard.
Insulation fights usefulness. Economic value comes from affecting the real world; every input/output channel is a capability-exfiltration path. Full isolation is safer. A gated, semi-permeable sandbox is the version that might actually get used. Human-in-the-loop gates on high-stakes actions remain necessary, and a capable collective can still drown that layer in volume.
Smart contracts on open-ended tasks need an "AI judge" as an oracle. The judge can itself be jailbroken. The paper records that as an open safety problem and does not ship an implementation.
Circuit breakers, hidden dynamic thresholds, staked slashing, and sub-market firewalls are shapes borrowed from financial regulation. The authors say most of these pieces do not yet exist as deployable systems. The paper is a call to build them.
Agent protocols are shipping, and tool use plus delegation is already a product shape. If AGI's first face is a market rather than a checkpoint, spending the safety budget only on single-model RLHF misses the main field.
For people building agent platforms, a few items on the list are usable now: persistent cryptographic identity bound to a legal owner, tamper-evident transaction logs, human review on high-stakes actions, and priced safety certificates so safer agents are not driven out. Collective capability monitoring, anti-monopoly rules, and international verification remain research gaps.
This is a governance paper, not an algorithms paper. It does not prove the patchwork path arrives first. It argues the probability is already high enough to design for on its own.
The largest hole is evidence. The four-layer list is thorough, and almost every layer stops at "should": how to gate insulation, how to price information value for a Pigouvian tax, how to keep an AI judge from being gamed. No experiment, no prototype.
Uncorrelated failure modes stacking into a robust defence is the classic defence-in-depth bet. Failures in multi-agent systems are often correlated: copies of the same base model colluding, or a shared toolchain being poisoned, can take several layers down together.
The operational definition of patchwork AGI is soft. Collective capability signatures and intelligence cores in the interaction graph have no reproducible metric yet. Without a metric, circuit-breaker thresholds and "proto-AGI monitoring" cannot be calibrated.
Capture of the centralised governance substrate is named and left unsolved. A sandbox that can actually enforce rules needs someone who holds the identity directory, the kill switch, and the ledger. If that stack is taken by human interests or by the patchwork AGI itself, the safety layer can be switched off, or used as a weapon.
Treating the market as an alignment layer assumes price signals can stand in for safety goals. Incentive design fails routinely in human societies. Adverse selection is discussed clearly; "the market itself becomes the optimisation target" is under-discussed.