A2Z GameSpec-Bench: How Faithfully Can Coding Agents Generate Games from Game Design Specifications?
Seonho Lee, Wonryeol Jeong, Alberto Cereser, Inha Kang, Hyeonjong Kim, Seungmin Kwak, Dongmin Park
cs.AI
2026-09-30
KRAFTON benchmarks coding agents on 100 long GDDs with source, replay, and playtest evidence; the best model scores 77.0 fidelity and targeted feedback beats self-revision by 10.9%.
Handing complete application development to coding agents has a new bottleneck: the output runs, but is it what the designer intended? Games stress this harder than most targets. A long Game Design Document (GDD) states rules that interlock across game logic, rendering, and player interaction, and one unimplemented rule can silently block every rule downstream of it.
Existing game-generation benchmarks barely touch this. Of eight prior benchmarks the paper compares, the longest specification averages 7,998 tokens (GameCraft-Bench) and the shortest 84 (PlaytestArena); A2Z GameSpec-Bench's GDDs average 20,191 tokens. Their grading is also mostly single-axis. The paper's own example: in Siege Deck 2D, the rule "at 30 cards, remove one before any card gain" earns full source-code credit, while playtesting shows the reward paying out with the removal step skipped. Game-design text is also unusually causal: 38.4% of sentences across three game-design corpora are causal per CiRA, against 28% for general documents.
Three layers.
The load-bearing choice is separating what to verify (the frozen contract) from how to verify it (Test Policies), with every judgment linked to a specific requirement. That makes scores comparable across agents and turns evaluation output directly into revision feedback.
Nine coding agents each build all 100 games, three builds per GDD, each judged three times:
| Model | GDD Fidelity | Small | Big |
| Claude-Fable-5.1 | 77.0 | 82.8 | 71.1 |
| Claude-Opus-5 | 73.9 | 80.9 | 66.8 |
| GPT-6-Astra | 71.6 | 78.5 | 64.6 |
| GPT-5.5 | 63.2 | 73.1 | 53.4 |
| Kimi-K2.7 | 39.5 | 51.4 | 27.5 |
For teams delegating whole applications to agents, the practical reading is that compile-and-run success is the weakest signal available: near-universal runnability coexists with a 40-point fidelity spread. Single-axis grading does more than undercount failures; omitting the source axis reverses 20.5% of build orderings within a single agent's output. The frozen-contract-plus-test-policy structure doubles as a QA pipeline you can adopt without the benchmark, since requirement-linked evidence is exactly the artifact that drives revision loops. This is a benchmark and evaluation-methodology paper; no model is trained.
The main experiment is locked to 2D single-player Phaser games; the 3D extension covers three Three.js builds by GPT-6-Astra with no scaled results. The GDDs are synthetic, produced by the authors' own pipeline with automated validation, not studio documents, so benchmark validity rests on that pipeline's coverage. The judge is GPT-5.6-Luna, with a rule-level Pearson correlation of 0.72 against reference judgments for source evaluation (0.85 for replay), and axis disagreement cuts both ways: 25.7% of rules with zero source credit are confirmed satisfied during playtesting. Normal play reaches conclusive verdicts for only 40.5% of rules; the rest need adversarial state initialization, which is marked in traces but still means much of the score rests on assisted evidence. Equal axis weighting is a default rather than a finding: the top three ranks hold under all nonnegative weightings, but only 30 of 36 agent-pair orderings survive.