FULL STORY

Astra Plays Chess: From Losses to Comeback

Mike Frank put OpenAI's Astra through days of chess matches, losing to lower-rated bots before improving his engine setup and finally beating the 1800-rated Wally bot.

2026-09-06 ~ 2026-09-08 · 4 episodes · 20 posts

Episode 1 · Astra Loses to 1300-Elo Chess Bot, Then Wins with Ultra Mode and Visualization Tool (2026-09-06, 7 posts)

On 09-06, MikePFrank posted a multi-part experiment letting AI model Astra play chess directly against chess.com's Nelson bot (ELO 1300, intermediate). Unaided, Astra blundered repeatedly and resigned; even at Extra High reasoning it still lost. But switched to Ultra mode and armed with a self-built single-step board visualization tool, Astra dominated Nelson and won. The series exposes frontier models' weakness in long-horizon strategy while showing how coding and agent abilities can compensate.

Confirmed

  • At High reasoning, unaided, Astra kept pace early but made multiple obvious blunders and resigned (m1, m6, m7).
  • Retried at Extra High: slightly better, still lost to Nelson (m3, m4, m7).
  • No external engine cheating (m1).
  • After the game Astra wrote a browser-based game replay app on the spot; the author also had it add a 'single-step board visualization' capability to avoid low-level mistakes (m3, m4).
  • Turning point: Astra (Ultra) with the visualization tool completely dominated Nelson, whose position collapsed, and won; a full replay link was shared (m5, m6).
  • The author noted LLMs have always been weak at out-of-the-box chess, but modern models can write complete chess engines—letting Astra try compensating via code (m2).

Why it matters

  • The series vividly shows frontier models' shortcomings in long-horizon strategic reasoning: even at Extra High, they lose to a 1300-Elo intermediate bot.
  • But the ending isn't just a flop: Ultra mode plus self-installed tooling reversed the result, showing an alternative path—models can write engines and visualization tools on the fly, using coding and agent abilities to offset native reasoning limits, a classic 'agentic AI' play.
  • It also shows that the same model's capability can vary greatly across reasoning tiers and tool setups; a single game is insufficient to judge a model.

Episode 2 · GPT-6 Astra beats chess engine with raw model output, 41-move mate (2026-09-06, 3 posts)

Mike Frank reports what may be the first time an LLM—GPT-6 Astra—defeated a real chess engine using pure model output, no hidden engine or special harness, delivering a 41-move checkmate verified against the original game record.

Episode 3 · OpenAI's Astra Loses Chess Match to 1800-Elo Bot (2026-09-06, 7 posts)

Mike Frank posted live updates from September 6 to 8, streaming games in which OpenAI's model Astra (Ultra configuration, equipped with a chessboard visualization tool) faced off against Wally, an ELO 1800 bot on chess.com, and ultimately lost—demonstrating the capability ceiling of frontier models on chess tasks.

Confirmed

  • The same Astra configuration had previously beaten an ELO 1500 bot with ease; Mike Frank intended to probe its limits, and it fell short at 1800. The first half of the game was closely contested
  • Key turning point: Astra retreated its bishop to d3, and Mike Frank immediately predicted the bishop would get trapped; Wally then advanced its b-pawn, traded pawns on b5, and pushed its c-pawn to c4, creating a direct capture threat on the bishop, which had no escape and was officially lost
  • In his post-game review, Mike Frank noted that Astra's early-position judgment and engine analysis were inadequate, and that it recognized this too late; after losing the piece, Astra briefly regained material, went up a pawn, and hoped for promotion, but the position had become too passive—in the end, after a series of checks from Wally, it concluded it could not escape checkmate and resigned

Why it matters

  • Astra is OpenAI's new model, and its earlier win against the 1500-rated bot was decent; this game vividly exposes the model's stability weaknesses on a seemingly simple chess task: poor opening judgment combined with a lack of foresight about how the position would develop made the game ultimately unrecoverable
  • This kind of public, step-by-step streamed post-mortem—from predicting the trapped bishop to the prediction coming true, to the resignation—offers a concrete case study of the reasoning and planning limits of frontier AI models

Episode 4 · AI agent Astra defeats 1800-rated chess bot after engine improvements (2026-09-08, 3 posts)

Mike Frank's AI agent Astra defeated the 1800-rated Wally bot in 31 moves after he improved its self-written Python engine and adopted a one-hour clock. The review shows Astra had calculated a two-move mate as early as move 29.