FULL STORY

Astra Plays Chess: From Losing to Bots to Beating Them

Mike Frank's days-long chess experiments saw Astra go from losing to a 1300-Elo bot to exceeding 1800 Elo without any engine, charting rapid gains.

2026-09-06 ~ 2026-09-08 · 5 episodes · 28 posts

Episode 1 · Astra Loses to 1300-Elo Chess Bot, Then Wins with Ultra Mode and Visualization Tool (2026-09-06, 7 posts)

On 09-06, MikePFrank posted a multi-part experiment letting AI model Astra play chess directly against chess.com's Nelson bot (ELO 1300, intermediate). Unaided, Astra blundered repeatedly and resigned; even at Extra High reasoning it still lost. But switched to Ultra mode and armed with a self-built single-step board visualization tool, Astra dominated Nelson and won. The series exposes frontier models' weakness in long-horizon strategy while showing how coding and agent abilities can compensate.

Confirmed

  • At High reasoning, unaided, Astra kept pace early but made multiple obvious blunders and resigned (m1, m6, m7).
  • Retried at Extra High: slightly better, still lost to Nelson (m3, m4, m7).
  • No external engine cheating (m1).
  • After the game Astra wrote a browser-based game replay app on the spot; the author also had it add a 'single-step board visualization' capability to avoid low-level mistakes (m3, m4).
  • Turning point: Astra (Ultra) with the visualization tool completely dominated Nelson, whose position collapsed, and won; a full replay link was shared (m5, m6).
  • The author noted LLMs have always been weak at out-of-the-box chess, but modern models can write complete chess engines—letting Astra try compensating via code (m2).

Why it matters

  • The series vividly shows frontier models' shortcomings in long-horizon strategic reasoning: even at Extra High, they lose to a 1300-Elo intermediate bot.
  • But the ending isn't just a flop: Ultra mode plus self-installed tooling reversed the result, showing an alternative path—models can write engines and visualization tools on the fly, using coding and agent abilities to offset native reasoning limits, a classic 'agentic AI' play.
  • It also shows that the same model's capability can vary greatly across reasoning tiers and tool setups; a single game is insufficient to judge a model.

Episode 2 · GPT-6 Astra beats chess engine with raw model output, 41-move mate (2026-09-06, 3 posts)

Mike Frank reports what may be the first time an LLM—GPT-6 Astra—defeated a real chess engine using pure model output, no hidden engine or special harness, delivering a 41-move checkmate verified against the original game record.

Episode 3 · OpenAI's Astra Loses to 1800-Elo Chess Bot After Trapped Bishop Blunder (2026-09-06, 7 posts)

From September 6–8, Mike Frank live-blogged OpenAI's new model Astra (Ultra setup with a board visualization tool) playing chess against Wally, an 1800-Elo bot on chess.com. Astra ultimately lost, illustrating the capability ceiling of frontier models on board games.

Confirmed

  • The same Astra configuration had easily beaten a 1500-Elo bot; Frank's goal was to probe its upper limit, and it failed at 1800, though the game stayed close through the first half
  • Key turning point: Astra retreated its bishop to d3, and Frank immediately predicted it would be trapped. Wally advanced the b-pawn, traded on b5, then pushed the c-pawn to c4, creating a direct capture threat—the bishop had no escape and was duly lost
  • Frank's postmortem: Astra's early engine analysis was inadequate and it realized this too late; after losing the piece it briefly regained material and held an extra pawn hoping to promote, but the position was already too passive
  • Endgame: after a series of checks from Wally, Astra judged it could not escape the coming checkmate and resigned

Why it matters

  • The game exposes a stability weakness of frontier models on seemingly simple board tasks: poor early evaluation plus lack of anticipation of tactical developments proved unrecoverable
  • The step-by-step public live documentation—from the trapped-bishop prediction to its confirmation and the resignation—offers a concrete case study of frontier AI reasoning and planning limits

Episode 4 · Astra Scores Over 1800 Elo in Chess Without Engines (2026-09-07, 5 posts)

Google Astra's chess ability drew attention: on 09-07 thomasahle asked about its strength, and ivenger cited existing evaluations putting it around Elo 1500-1800. On 09-08 Mike Frank reported hands-on testing: to prevent "cheating" (downloading the Stockfish engine or chess databases), he verified the model could play without external tools, and its measured rating exceeded 1800.

Confirmed

  • 09-07: thomasahle asked about Astra's chess level; ivenger replied existing evaluations put it at Elo 1500-1800.
  • 09-08: Mike Frank measured Astra at over 1800 Elo with Stockfish and game databases disallowed.
  • Mike Frank also noted that in the earlier tile-puzzle case, Astra could have written its own solver within minutes, suggesting its real capability may be underestimated.

Controversy

  • iruletheworldmo questioned the value of chess benchmarks: a model could simply download Stockfish and beat Magnus Carlsen. Yet the benchmark remains watched precisely because it tests the model's own reasoning, which is why Frank emphasized the no-tools setup.

Why it matters

  • Chess has become a representative test of frontier models' general reasoning, but methodology—whether external tools are disabled—determines credibility. This tool-free measurement gives a closer estimate of the model's true strength.

Episode 5 · AI Agent Astra Revamps Engine, Checkmates 1800-Elo Chess Bot in 31 Moves (2026-09-08, 6 posts)

Mike Frank's AI agent Astra got its revenge in a human-vs-AI chess match: after switching to a self-built engine under relaxed time controls, Astra checkmated the 1800-rated Wally bot in 31 moves as Black, spending only about 6 minutes more than its opponent. Frank now believes its strength clearly exceeds 1800 Elo. The case shows how a general-purpose agent, with a self-written engine and deliberate planning, can approach stronger chess AIs, and the setup is slated for open-sourcing.

Confirmed

  • Astra had lost repeatedly to Wally; after several engine iterations Frank released Astra (Ultra) v0.2 with a home-grown Python game engine
  • Conditions were relaxed to a 1-hour clock with flexible move timing; Astra won by checkmate playing Black (0–1)
  • Mid-game Astra played a queen-rook fork, reaching a +20 evaluation
  • The engine found a mate-in-two at move 29; the actual checkmate came at move 31, and Astra used only 6 minutes more than the opponent overall
  • Frank published an interactive game-9 review (hosted on netlify), for the first time including the engine's per-move evaluations; the game-10 replay is also public
  • Asked by thomasahle about the framework, Frank said the setup is Codex plus custom tools and skills, and he plans to open-source it on GitHub

Unconfirmed

  • "Over 1800 Elo" is the author's personal estimate from play; Astra's exact rating is unmeasured, and the open-source plan has not yet materialized

Why it matters

  • The case shows a general AI agent can write and iterate its own game engine rather than rely on an off-the-shelf chess engine, reaching near-1800-human strength under relaxed clocks via deliberation
  • The published per-move engine-eval review gives the community verifiable material to audit Astra's real decision process; if open-sourced, the "Codex + custom tools" recipe can be reproduced and extended