SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn Reinforcement Learning
Bo Liu, Leon Guertler, Simon Yu, Zichen Liu, Penghui Qi, Daniel Balcells, Mickel Liu, Cheston Tan, Weiyan Shi, Min Lin, Wee Sun Lee, Natasha Jaques
ICLR 2026
cs.AI, cs.CL, cs.LG
2025-07-01
SPIRAL trains by self-play on TicTacToe, Kuhn Poker, and negotiation; Qwen3-4B-Base rises from 34.0% to 44.5% on eight benches, beating SFT on 25k expert games.
Models such as o1 and DeepSeek-R1 learned long chain-of-thought from verifiable rewards. Someone still had to write the problems, the graders, and the domain rules. Self-play solved that bottleneck in Go and Dota: the opponent grows with you, so data never runs out. On language models the idea had mostly been single-turn, offline, or boxed into word games.
SPIRAL asks a blunt question. If a model plays itself in multi-turn zero-sum language games, do math and knowledge scores move, even though those benchmarks never appear in training?
Three two-player games from TextArena supply the environment: TicTacToe (spatial), Kuhn Poker (a three-card poker drill for probability), and Simple Negotiation (bargaining). One policy plays both seats, with a system prompt for the role. Reward is sparse: +1, 0, or -1 at the terminal state, nothing in between.
The optimizer is REINFORCE with a baseline. In a zero-sum game the two seats do not share an expected return. First-move edge and hidden cards both bias the mean. Role-conditioned Advantage Estimation (RAE) keeps an exponential-moving-average baseline per game and per seat, then subtracts it from the terminal return. Without RAE, thinking traces collapse from about 2,000 characters toward empty within roughly 100 policy iterations (the abstract says about 200 steps), and models start emitting boxed moves with no reasoning. General scores drop from 44% to 40%. With RAE, replies stay in the 1,300-1,500 character band and scores rise from 40% to 47%.
Training is online actor-learner: vLLM for sampling, TextArena for the games, Oat for distributed updates. 400 steps, 128 samples per step, eight H100s, learning rate 1e-6. Baselines include SFT on 25,000 expert trajectories from Qwen3-32B, plus frozen opponents (random, Mistral-Small-3, Gemini-2.0-Flash-Lite).
Multi-game SPIRAL lifts every family they tried, with a wide spread:
| Model | Base avg | SFT-Multi | SPIRAL-Multi |
| Qwen3-4B-Base | 34.0 | 39.7 | 44.5 (+10.5) |
| Qwen3-8B-Base | 39.5 | 46.1 | 49.6 (+10.1) |
| Octothinker-8B-Base | 25.8 | 27.0 | 33.8 (+8.0) |
| Llama-3.1-8B-Instruct | 23.9 | 25.0 | 25.9 (+2.0) |
| DeepSeek-R1-Distill-Qwen-7B | 60.4 | 58.3 | 61.8 (+1.4) |
On Qwen3-4B, AMC-23 moves from 42.4 to 61.6, AIME24 from 9.6 to 19.7, MATH500 from 73.4 to 78.2. The already-RLVR Distill-7B still ticks up; SFT on the same games pulls it down.
Fixed opponents saturate. Win rate versus Gemini climbs from 0% to 62.5%, a sign of an exploitable script. Self-play versus a 16-step-old copy stays near 50%-52%. Random opponents teach almost nothing.
GPT-4.1 labels 290 game traces and 46,792 math solutions. Case-by-case analysis transfers almost intact (72% to 71%). Pattern recognition amplifies (35% to 45%). Expected-value talk transfers more selectively (78% to 28%). Specialists win similar out-of-distribution games (poker specialist 91.7% on Pig Dice). A multi-game agent averages 59.5% against Gemini-2.0-Flash, above the best specialist at 52.9%.
Few public stacks run fully online, full-parameter, multi-turn self-play for LLMs. The code is out. For unlabeled reasoning RL, RAE is the piece to copy: seat asymmetry will collapse vanilla REINFORCE into empty thoughts.
This is a transfer study, not a new path to general intelligence. TicTacToe and Kuhn Poker are shallow. Instruct models and already-RLVR checkpoints move only one or two points.
No Limitations section. The game set is tiny, fully specified, and barely partially observable. An LLM tagging "this math solution used pattern X" is correlation, not a causal map. Eight reasoning benches lean hard on math; MMLU-Pro and GPQA gains do not license claims about coding or long tool use. Llama Instruct's +2.0 says aligned models have little room left. The acknowledgements mention a DARPA/AFRL contract unrelated to the experiments. The paper never asks whether adversarial game play teaches harmful tactics.