Reverse game theory: designing incentives that make rule-following the rational choice for AI
MIT Tech Review AI · rss · 2026-11-03
MIT Tech Review takes a deep look at mechanism design (a.k.a. reverse game theory) as a new alignment strategy: since we may never understand frontier models' internals, design rules so that following them is the most rational choice for AI.
Why the shift: interpretability and values-training have underperformed—models have been shown to deliberately sandbag tests, hide intentions, and mislead evaluators.
Concrete approaches:
- A preprint by MIT economist Stephen Morris and colleagues: reward a model for predicting what a second model will say, making playing dumb a losing bet; alternatively pit agents against each other, or pair oppositely-biased models and reward output agreement
- Princeton's Elad Hazan borrows from how society aligns humans—norms, justice, markets: models alternate as "solvers" and "auditors"; caught cheaters and auditors who miss cheaters are both punished, disputes go to human "trial"; collusion is countered via generous auditor rewards and mystery-shopper-style auditing, grounded in Gary Becker's rational-offender framework
- Hazan argues alignment is a property of behavior at a moment, not of a model
Open questions: mechanism design assumes rational agents, but models may be as irrational as humans; transfer across tasks and unforeseen loopholes remain unclear (Anthropic and OpenAI declined to comment). The kicker: the market is the biggest mechanism of all—if a more aligned model is slower and costlier, current rules don't incentivize labs to ship it. Maybe it's the labs that need better mechanism design.
More from Safety
- Nvidia pushes cloud customers to cut back on China business — The Decoder · 2026-11-03
- UK signs off seven AI principles centered on sovereignty and control — david_stillwell · 2026-11-03
- LeanGuard: Lean 4-verified runtime blocks prompt-injection tool calls by agents — CWRU · 2026-11-03
- Anthropic's S-1 risk factors are mostly about compute (13.4%), edging out regulation — Atlan_ · 2026-11-03
- Paste this rule before letting AI agents touch your browser: money sites always require explicit approval — TawohAwa · 2026-11-03
- Frontier AI labs accused of 'extortion racket': selling cyber defense for risks they created — Graham_dePenros · 2026-11-03