Brood War Bench Pits LLM Agents Against StarCraft — Every Model Loses to Novices

dotey · x · 2026-09-22

Developer Ben Swerdlow built Brood War Bench, letting general-purpose LLM agents directly play StarCraft: Brood War to see if they can build bases, train units, and fight on their own. Every model performed below novice level.

Highlights:

Codex Astra (rank 1): 18-0 record, but its best tactic is harassment — sending a single Probe to disrupt the enemy base, since opposing agents freeze for tens of seconds wondering what to do. Weak at actual economy and large battles, often throwing one or two units at the enemy instead of building up an army.
Claude Fable (3rd, 83.3% win rate): the most "serious" player — builds economy, climbs the tech tree, produced Mutalisks and researched Templar tech, though research often outpaced its army.
Grok 4.6 (worst): in a 43-minute match it emitted over 11,000 reasoning tokens but only 6 batches of commands and never built a single combat unit — playing real-time strategy like a turn-based game.

Key takeaway: current LLMs remain far from capable in environments demanding continuous observation, fast decisions, and multitask coordination — a stark contrast to DeepMind's AlphaStar, which was a purpose-trained RL system that beat pros in 2019.

Original post →

Brood War Bench Pits LLM Agents Against StarCraft — Every Model Loses to Novices

More from coding & agent