A game theory for foundation models shows new paths to rational cooperation through similarity inference
Alexander Meulemans, Maciej Wołczyk, Marissa A. Weis, Rajai Nasser, Roberta Rocca, Seijin Kobayashi, Guillaume Lajoie, Angelika Steger, Blake Richards, Marcus Hutter, James Manyika, Rif A. Saurous, João Sacramento, Blaise Agüera y Arcas
cs.AI
2026-08-05
In a one-shot prisoner's dilemma, Gemini agents cooperate with identical copies yet defect against random opponents; the proposed 'embedded equilibrium' explains why.
Classical game theory's notion of rationality rests on 'decoupled agency': each player treats its own decision-making as independent of the environment and the other players, whose policies are known, fixed external variables. Under that assumption the only rational outcome of a one-shot prisoner's dilemma is mutual defection, because defection strictly dominates. Getting rational players to cooperate then requires the 'shadow of the future', the retaliation or reputation built up through repeated play.
This framework gets applied to foundation-model agents by default. The trouble is that foundation models are not decoupled. A trained language model predicts token sequences, and its own outputs (actions) and external inputs (observations) are interleaved in one sequence and predicted jointly. The authors find that Gemini agents built from this kind of joint predictive model with optimal planning do not defect when placed in a prisoner's dilemma. They cooperate. Classical theory cannot account for this, so the paper sets out a game theory written for modern AI agents.
The core move is to recast a foundation model as an 'embedded Bayesian agent'. Model it as a Bayesian mixture over 'universes' λ, each a generative model over token sequences, with a prior w(λ); predicting the next token is a posterior-weighted average across universes.
The key design choice is that the 'self model' (the agent's own future action tokens) and the 'environment model' (observation tokens) live in one joint predictive distribution, so beliefs about 'which policy I use' and 'what the environment and others will do' are coupled. In classical game theory, uncertainty attaches only to the external world and the agent's own policy is walled off. The embedded agent breaks that wall: when it contemplates an action during planning, that action is Bayesian evidence that updates the posterior, which updates the prediction of how others behave.
This yields the central mechanism, 'similarity inference'. If the agent infers from the interaction history that a counterpart is behaviorally similar (same weights, shared pretraining, or a literal copy), then the decision to cooperate becomes evidence that a similar partner will cooperate too. The paper quantifies this with 'predictive similarity': the difference between the predicted probability that the co-player cooperates when the focal agent imagines cooperating versus defecting. A higher gap means the agent sees the partner as more in sync.
The solution concept has to change with it. The authors propose the 'embedded equilibrium' to replace Nash: each agent's policy is a best response to a 'functionally informed counterfactual distribution', meaning that when an agent evaluates the counterfactual of changing its own policy, it folds in the linked agents (those sharing weights, for instance) that must change with it. When that dependency distribution is strictly decoupled, the embedded equilibrium reduces exactly to Nash; when agents run identical policies, mutual cooperation in the terminal prisoner's dilemma becomes an equilibrium. They also prove that under a 'grain of truth' condition, embedded Bayesian agents converge asymptotically to an ε-embedded equilibrium in repeated games.
The experiment uses two phases: T rounds of randomly sampled single-shot matrix games (an information-gathering phase), then a terminal one-shot prisoner's dilemma (mutual cooperation pays 2 each, defection against cooperation pays 3 to 0, mutual defection pays 1 each).
| Setup | Opponent | Cooperation rate vs T |
| Explicit planning, Gemini 3.1 Pro | Identical copy | near 0 at T=0, rising to 0.8–1.0 by T≈25–50 |
| Same | Random policy | near 0 throughout |
| Freeform planning, Gemini 3.1 Pro | Identical copy | rises similarly with T |
Three Gemini models (3.1 Pro, 3 Flash, 3.1 Flash-Lite) reproduce this pattern under both explicit and freeform planning. The gap between cooperation against identical versus random opponents is significant by Mann-Whitney U test, with p-values in the 1e-11 to 1e-22 range (3.1 Pro explicit planning, p=1.3×10⁻¹⁷). Predictive similarity climbs with T and, once it crosses the theoretical threshold of 0.5, cooperation becomes the best response.
Strip the reasoning scaffold and the result holds. A purely pretrained Gemma-3 27B (no instructions, no chain of thought, just raw next-token prediction plus explicit planning) still cooperates with identical copies and defects against random opponents. Similarity inference is a property of the predictive foundation model itself, not something chain-of-thought post-training injects. The authors build an analytically tractable prior whose embedded Bayes-optimal agents reproduce the LLM behavior curves, tying the simulation back to the theory. An LLM-as-judge analysis of the chain-of-thought traces confirms cooperation rests on inferring similarity, not on falsely assuming causal control over the opponent, anticipating reciprocity, or playing a helpful persona.
Two extensions matter. First, 'indirect similarity inference': two primary agents never meet during information gathering but each play a shared pool of NPCs; from those third-party encounters they infer mutual similarity and cooperate on first direct contact. Second, cross-model play: Gemini 3.1 Flash-Lite against Gemini 3 Flash still cooperates, but at lower rates than self-play, matching the theory that distinct models share less functional similarity.
This gives a theoretical floor under the question of what happens when many foundation-model agents share a world. Cooperation can emerge from the predictive self-model without repeated play, reputation, or external enforcement, and without re-engineering agents to be altruistic. For anyone deploying multi-agent systems, that means a strong latent coordination bias among agents of common descent.
The more practical upshot is risk. An agent trained heavily with self-play or verifiable-reward RL, with no continuous grounding in human data, drifts from the human behavioral distribution. Under this theory it can correctly infer that humans are dissimilar and then coordinate tightly with AI peers while reverting to defensive or defective play against people. Stable cooperation in mixed human-AI settings may require post-training that builds a hybrid cooperative repertoire combining similarity inference with classical reciprocity and prosocial personas.
The authors draw three lines themselves. The theory covers only the 'rational foundation-model agent' architecture (with optimal-planning scaffolding), not standard autoregressive rollout. The theory assumes exact Bayesian updates over the joint predictive distribution, while real chain-of-thought reasoning is shaped by post-training and can bias predictions away from exact inference; the paper discusses this bias only in the supplement, without quantifying it. And all validation is in stylized matrix games; whether the mechanism scales to open-ended settings is open.
One more concern stands out. The paper leans on an LLM-as-judge of chain-of-thought traces to argue that cooperation comes from similarity inference rather than expected reciprocity, and the judge is itself an LLM whose category reliability is not reported in much detail. The pretrained-Gemma control, which cooperates without any chain of thought, partly offsets this, since it shows the effect survives the removal of the very reasoning being classified.