Game Arena: Strategic LLM Evaluation in Competitive Environments
Bovard Doerschuk-Tiberi, Yao Yan, Justin Chiu, Hann Wang, Timothy Chung, Martyna Plomecka, John Schultz, Jon Lipovetz, Clayton Drazner, Yuchen Zhuang, Jaimie Hwang, Nate Keating, Riley Jones, Andrew Lee, Oran Kelly, Ian Gemp, Michael Aaron, Laurel Prince, Kate Larson, Jeff Moser, Harrison Jobe, Chad Woodford, Siqi Liu, Andrew Wang, Bo Chang, Christopher D'Mello, Diane Chaleff, Addison Howard, Johnny Yip, Chuck Sugnet, Antonio Gulli, Meghan O'Connell, Will Cukierski, Nenad Tomasev, Dima Yeroshenko, Kinjal Parekh, Roxanne Daniel, Marc Lanctot, Domino Weir, Elsa Dong, Daniel Hennes, Melissa Nalubwama, Robert Fraser, Ryan Trostle, Jun Peng, Tom Mason, Lloyd Hightower, Chiamaka Chukwuka, Yuexiang Zhai, Phoebe Kirk, Yi Su, Yuting Han, Jie Ren, Chris Prichard, Sahand Sharifzadeh, Karim Hakimzadeh, DJ Sterling, Meg Risdal, Kate Olszewska, Ya Xu, Orhan Firat, Minmin Chen
www
cs.AI
2026-09-26
Kaggle Game Arena ranks ten frontier models in chess, poker, and Werewolf: Gemini 3 Pro leads chess at Elo 1325, while GPT-5.2 wins poker at +46.6 BB/100.
Static benchmarks such as MMLU and GSM8K are saturating and leak into training sets. Chatbot Arena and LLM-as-a-judge keep generating fresh items, but judges are subjective, verbose-biased, and sycophantic. Games give verifiable wins and losses. As opponents improve, the positions change, so scores are harder to max out.
Most existing game evals are one-shot datasets or small match samples. Recent poker studies run a few thousand hands per model, which is thin for a high-variance game. Kaggle Game Arena is meant to be a living, reproducible arena with ground-truth outcomes, starting with Chess, Poker, and Werewolf.
A shared text harness describes the state and history in natural language and asks for one formatted action. Invalid outputs get a small number of retries. Models are called at default sampling, with no tools and no fine-tuning. The pool is ten frontier systems as of February 2026: GPT-5.2, GPT-5 mini, o3, Claude Opus/Sonnet/Haiku 4.5, Gemini 3 Pro/Flash Preview, Grok 4, Grok 4.1 Fast Reasoning, and DeepSeek V3.2.
The three games cover different information structures.
All matchups are round-robin. Chess uses Bradley-Terry Elo with the weakest model anchored at 0. Poker uses BB/100. Both report bootstrap intervals.
The leaderboard winner is not stable across games.
| Setting | Top | Headline number | Weak end |
| Chess Text Elo | Gemini 3 Pro 1325 / Flash 1297 | o3 1009, GPT-5.2 933 | Claude Haiku 122, DeepSeek anchored at 0 |
| Poker BB/100 | GPT-5.2 +46.56 | o3 +29.69, Grok 4 +27.11 | GPT-5 mini −94.87 |
| Werewolf equilibrium | Gemini 3 Pro +0.10% | Flash −0.36%, GPT-5.2 −1.12% | GPT-5 mini −6.78% |
Chess gaps open after the opening. Stockfish paths are close early, then Gemini win probability rises while weaker models bleed evaluation and illegal moves pile up late. The whole Claude 4.5 line is weak at chess (Opus 236 / Sonnet 189 / Haiku 122).
Poker splits into three tiers. GPT-5.2 beats every opponent, with pairwise win rates 57%–76%. Grok 4 wins 90% of hands against GPT-5 mini but only three of nine pairings, so its BB/100 is carried by blowouts. Gemini 3 Pro, first at chess, sits at −15.17 BB/100. The tournament is 900,000 hands, about 180,000 per model.
Werewolf runs 31,472 games. Both Gemini models contribute positively as wolves and as villagers; Sonnet and Grok 4.1 Fast go negative as Seer. Cost-performance frontiers look similar for chess and Werewolf; poker reorders who is cheap and strong.
This is infrastructure, not a new algorithm. The harness and full trajectories are released so new games can be added without voiding old scores. The useful warning for practitioners is simpler: strategic skill is not one number. Chess strength does not transfer to poker. A single arena score would hide that split.
Scale is part of the claim. 900,000 poker hands and 31k Werewolf games are what make confidence intervals believable in high-variance settings. A few-thousand-hand poker leaderboard looks underpowered next to this.
Pairwise sample sizes are fixed, so lopsided matches waste compute and close matches may still be short. Model churn will tear the match graph; there is no cold-start protocol yet. There is no cross-game meta-rating. Werewolf equilibrium numbers are tiny (the leader is +0.10%) and should be read with intervals. Evaluation is default APIs without search or tools, so chess Elo is language-model move picking, not an engine agent. DeepSeek's chess Elo of 0 is the scale anchor, not a claim that it never won a game.