Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Simonyan, Demis Hassabis
cs.AI, cs.LG
2017-12-06
Given only the rules, AlphaZero beats Stockfish in 4 hours (28-72-0 in 100 games) and Elmo in under 2 hours, with one algorithm for chess, shogi, and Go.
Computer chess spent decades on one architecture: a handcrafted evaluation, tuned by masters, sitting on a heavily adapted alpha-beta search. Stockfish, the 2016 TCEC champion, is that design at its peak: opening books, endgame tablebases, quiescence search, null-move pruning, history heuristics. Shogi is harder still. The board is larger, and captured pieces can be dropped back anywhere. Elmo, the 2017 CSA champion, had only just started beating human title holders.
Those engines do not transfer. AlphaGo Zero had already shown that a deep net plus self-play can reach superhuman Go without human games. Go is a friendly domain for convolutions: translation-invariant rules, local liberties, eightfold symmetry, binary outcomes. Chess and shogi are not. Rules depend on rank, pieces fly across the board, and draws are common; chess is widely believed to be a draw with perfect play. The question is whether one tabula rasa algorithm, with almost no game-specific knobs, can master all three.
A single network \(f\theta(s)\) maps a board to a move prior \(\mathbf{p}\) and a scalar value \(v\). The value is an expected outcome, not a win probability: \(-1\) / \(0\) / \(+1\) for loss, draw, win. Search is generic MCTS. Each simulation walks from the root to a leaf, preferring moves with low visit count, high prior, and high value, evaluates the leaf with the net, and backs up. The visit distribution \(\boldsymbol{\pi}\) at the root becomes a stronger policy target.
Training is self-play from random weights. Both sides move by MCTS. After the terminal score \(z\) is known, the loss is mean-squared value error plus policy cross-entropy plus L2:
\[l = (z-v)^2 - \boldsymbol{\pi}^\top\log\mathbf{p} + c\|\theta\|^2\]
Four changes strip Go-specific tricks out of AlphaGo Zero. The value target handles draws. There is no 8-fold symmetry augmentation, and MCTS does not randomly rotate the board. There is no "best player" gate that waits for a 55% win margin; one network is updated continuously and self-play always uses the latest weights. The same hyperparameters are reused across games. The only exception is Dirichlet noise at the root, scaled by typical branching factor: \(\alpha=0.3\) chess, \(0.15\) shogi, \(0.03\) Go.
Inputs encode only what the rules require. Chess uses an \(8\times8\times119\) stack (8-step history plus castling, repetitions, fifty-move counter) and an \(8\times8\times73\) policy over 4,672 move slots. Shogi uses 362 input planes and a \(9\times9\times139\) policy. Go keeps the AlphaGo Zero \(19\times19+1\) head. Illegal moves are masked. Training searches use 800 MCTS simulations.
Compute is large: 5,000 first-generation TPUs generate games, 64 second-generation TPUs train the nets. Each game is trained for 700,000 steps with batch 4,096. Chess ran 44 million games in 9 hours, shogi 24 million in 12 hours, Go 21 million in 34 hours.
On a 1-second-per-move Elo curve, AlphaZero passed Stockfish 8 after 4 hours (300k steps), Elmo in under 2 hours (110k steps), and AlphaGo Lee after 8 hours (165k steps). A footnote is explicit: AlphaGo Master and AlphaGo Zero were later trained about 100 times longer; that run is not repeated here.
Head-to-head matches used 1 minute per move and 100 games. AlphaZero and the 3-day AlphaGo Zero ran on one machine with 4 TPUs. Stockfish 8 and Elmo used 64 threads and a 1 GB hash.
| Matchup | Win | Draw | Loss |
| AlphaZero White vs Stockfish | 25 | 25 | 0 |
| AlphaZero Black vs Stockfish | 3 | 47 | 0 |
| AlphaZero White vs Elmo | 43 | 2 | 5 |
| AlphaZero Black vs Elmo | 47 | 0 | 3 |
| AlphaZero White vs AG0 (3-day) | 31 | n/a | 19 |
| AlphaZero Black vs AG0 (3-day) | 29 | n/a | 21 |
Totals: 28-72-0 against Stockfish, 90-2-8 against Elmo, 60-40 against the 3-day AlphaGo Zero.
Search is three orders of magnitude thinner. AlphaZero looks at 80k positions per second in chess versus 70 million for Stockfish, and 40k versus 35 million in shogi. The net concentrates the tree on a few promising lines. As thinking time grows, its Elo scales more steeply than either alpha-beta engine.
The 12 most common human openings (each played more than 100,000 times on 365chess) all appear on their own during self-play. Restarting from those openings, AlphaZero scored 242-353-5 as White (40.3% / 58.8% / 0.8%) and 48-533-19 as Black (8.0% / 88.8% / 3.2%) against Stockfish.
This 2017 preprint is the paper that took "self-play + deep net + MCTS" off the Go board and onto the classical games. MuZero, NNUE in engines, and most modern game AIs sit on this template.
The loop is short enough to reuse: the net proposes a prior and a value, MCTS turns them into a better \(\boldsymbol{\pi}\), then \(\boldsymbol{\pi}\) and the terminal \(z\) train the net. No handcrafted eval, no drawer of alpha-beta heuristics.
It is not a free lunch. Few labs can spend 5,000 TPUs on self-play. "No domain knowledge except the rules" still includes a grid-shaped encoder, legal-move generation, terminal scoring, and a max-length cutoff that declares a draw. Environments whose rules are hard to encode still need extra work.
The paper flags three issues itself. None of the classical chess enhancements are used, and some would likely add Elo. The full AlphaGo Zero training budget is not matched. High-level chess is draw-heavy, which compresses Elo.
The match conditions drew later criticism. A 1 GB hash is small for Stockfish 8 in 2017. Opening books and tablebases are not clearly specified for the baseline. 64 CPU threads versus 4 TPUs is not an apples-to-apples hardware comparison. One hundred games can rank two players; they cannot pin down a precise Elo gap. The later Science version added more games and time controls, and the numbers are not identical to this preprint.
The value head fits \(\{-1,0,+1\}\) with squared error. In drawish chess endgames the signal is thin. Endgame accuracy is not reported separately.