Debate Erupts Over Fully Self-Trained StarCraft AI as AGI Benchmark
A debate broke out on 09-26 between former DeepMind/Google researcher syhw (François Chollet's team) and Wisconsin professor Kangwook Lee over a StarCraft AI trained purely via self-play from scratch with zero human data, focusing on its ceiling and whether StarCraft qualifies as an AGI-related benchmark.
Confirmed
- syhw praised the AI's approach as the "right path" for game AI.
- His claim: with peak APM capped at 500 and average at 300 (near human level) plus 10x more training compute, the model would be even stronger.
- One side argued the model already shows near top-human performance, supporting its use as an AGI benchmark.
Unconfirmed
- Kangwook Lee stated there is no evidence any actual model can play SC1/2 at semi-pro level under strict API constraints, with or without behavior cloning (BC) bootstrapping.
- He expressed skepticism about AlphaStar and recent agents under such conditions.
- Whether self-play can break its ceiling with more compute lacks evidence, per Lee.
Why it matters
- The core dispute: AI StarCraft play can always be dismissed as "not real skill" — syhw noted the AI would be questioned for not using movement restrictions, making benchmark fairness itself contested.
- The debate reflects a long-standing standards war between game AI and AGI evaluation: APM caps, API constraints, and human-data usage directly determine whether a result counts as proof of intelligence.
2026-09-26 ~ 2026-09-26 · 5 related posts
Primary sources
- Self-Play Game AI Trained From Scratch Is 'The Game', Says Researcher; 10x Compute Would Make It Even Better — syhw ·
- Prof. Kangwook Lee: no SC/SC2 agent hits semi-pro level under hard API limits without micro-cheating — Kangwook_Lee ·
- StarCraft AI trained purely by self-play sparks AGI benchmark debate — syhw ·
- [source] Self-Play Game AI Trained From Scratch Is 'The Game', Says Researcher; 10x Compute Would Make It Even Better — syhw · 2026-09-26
- [source] StarCraft AI trained purely by self-play sparks AGI benchmark debate — syhw · 2026-09-26
- Researchers split on whether scaled self-play with capped APM can break model ceilings — Kangwook_Lee · 2026-09-26
- Professor with 30 years of StarCraft clashes with Google researcher over whether StarCraft is an AGI benchmark — Kangwook_Lee · 2026-09-26
- [source] Prof. Kangwook Lee: no SC/SC2 agent hits semi-pro level under hard API limits without micro-cheating — Kangwook_Lee · 2026-09-26