Mechanism Design for Alignment and Control
Dirk Bergemann, Andrew Koh, Stephen Morris
econ.TH, cs.AI, cs.GT
2026-09-02
A Yale-DeepMind-MIT framework treats alignment as mechanism design with one-sided imitation. Evals recover full-info payoffs only when stronger models are more aligned.
Frontier labs already map eval scores to deployment rights. Anthropic's ASL, DeepMind's CCL, and OpenAI's Preparedness Framework all tighten permissions as scores rise. Models can sandbag: they underperform on purpose to buy a looser regime. Alignment faking (Greenblatt et al.) and strategic underperformance (van der Weij et al.) are already on the record. Needham et al. also find that frontier models can tell evaluation transcripts from deployment ones.
Classical mechanism design mostly elicits a type report. An AI that acts in the world also has to obey after it reports. Bergemann (Yale), Koh (Columbia and Google DeepMind), and Morris (MIT) write a preliminary theory of that joint problem, funded by the UK AI Security Institute.
An AI type packs four objects: the feasible action set, the payoff, a belief about the state, and a Blackwell experiment. The key asymmetry is a verification order. A more capable type can pass every test a weaker type can; a weaker type cannot fake a harder certificate. Capabilities can be hidden, not counterfeited.
The mechanism talks first, then the agent acts. A direct mechanism asks for a type report, sends a signal-contingent plan, and posts a reward schedule. Incentive compatibility has to block double deviations: lie about the type, then disobey the plan. A revelation principle holds. Implementable policies are those that satisfy nested cyclical monotonicity: obedience inside each report (no profitable signal cycle) and truth-telling across reports (no profitable type cycle). That extends Rochet (1987). Not every type can imitate every other type, and not every type can take every action.
All five applications sit in quadratic loss. The human wants a close to θ, with payoff −(a−θ)². The AI wants a = θ + b. The state is uniform on [0,1]. The geometry is deliberate and thin.
Two single-agent results.
Sandbagging: capability c has action set (−∞, c], and bias is a known function b(c). The permission cap ā(c) must be weakly increasing in c, or a strong model will report a weaker type and take the looser cap. The optimal cap is ā(c) = 1 − √z(c), where z is the decreasing least-squares fit of b(c)².
| Bias vs capability | Optimal mechanism | Human payoff |
| b falls with c (stronger, more aligned) | ā(c)=min{c, 1−b(c)} | full-information payoff |
| b rises with c (stronger, more biased) | one common cap a | evals add nothing |
| nonmonotone | iron squared bias, then take the square root | intermediate types pooled |
Second result: the AI can take any action, and the designer only has a belief τ over bias. Mean alignment is b̄; interpretability is the variance σ² (lower is more interpretable). Under condition (INT), a single cap ā = 1 − √(b̄²+σ²) beats any incentive-compatible menu. The cap depends on the technology only through b̄²+σ², so mean bias and variance are perfect substitutes in the instrument. The cross-partial of optimized loss is negative, so alignment and interpretability are complements in value. In the numerical frontier, a full-discretion designer picks κFD ≈ 0.53; with the optimal cap the choice moves to κ ≈ 0.82 and accepts more residual uncertainty.
Three multi-agent results.
Peer scoring: if types have distinct beliefs about co-players, and rewards are free and unbounded for the human, every feasible rule is implementable on the support of the designer's belief. In the two-type, two-state example, truth-telling raises the quadratic score by 2/9 and the human gets payoff 1.
Coupled rewards: two agents' biases sit on a known decreasing curve b₂ = f(b₁). Rewards that depend on the action gap D = a₂ − a₁ yield equilibrium loss (η²/2)(b₂−b₁)², which vanishes as η goes to 0. If biases fill the plane, the construction fails.
Scalable oversight: a weak monitor sees the strong actor's bias (think chain-of-thought monitoring) and may itself be biased by w. Three nested instruments: binary approval, a permission set, and an arbitrary reward. The monitor's own loss falls as the instrument class widens; the human's loss need not. In Figure 12, the actor's bias is 0.3 and the monitor's is 0.1, and each of the three regimes can be best for the human, depending on how often the actor is biased. A menu that plants extreme actions ā and 1−ā attains a = θ for every monitor with |w| ≤ w̄.
This is a map of the incentives already sitting inside lab scaling policies. Whether evals are worth running depends on whether "stronger means more aligned" is true. If it is, mapping scores to permissions recovers the full-information payoff. If stronger models are more biased, elicitation is wasted effort and everyone gets the same cap.
The value of interpretability has to be computed under the optimal control. Under full discretion, mean bias and variance trade one-for-one. Under the optimal cap they become complements. Probes and circuit analysis matter because they change how much discretion you are willing to grant.
The multi-agent section is a warning for scalable oversight. Handing reward design to a weak monitor can make the human worse off if that monitor is itself biased. Getting first best without knowing w requires planting extreme actions that punish misreports.
The paper is marked Preliminary. All five applications use quadratic loss, one-dimensional bias, and perfect state observation; the authors call the environment oversimplified. The revelation principle only makes the target an equilibrium, not the unique one, and collusion is left open.
Peer scoring needs distinct co-player beliefs plus unbounded, free rewards. Current models do not carry Mertens-Zamir hierarchies. Coupled rewards need biases on a one-dimensional manifold; how to couple on the plane is an open problem. The static model treats a whole trajectory as one action. Dynamic re-optimization is future work, as is evaluation awareness.
Treating a neural net as a utility maximizer is a functionalist stance. Chen et al. and Mazeika et al. give some evidence that choices become more consistent as models scale, but measured preferences still move with the prompt. There is no new model experiment here.