GameXpert-Bench: How Far Are Coding Agents from Expert Game Development?
Kun Chen, Haorong Hong, Peizhong Gao, Jianfeng Lin, Tongxu Luo, Yuxuan Xie, Chenxu Liu, Jieling He, Zhongyuan Liu, Zeno Zeng
cs.AI, cs.CL
2026-08-22
GameXpert scores agents on 97 games, 100 multi-bug repairs, and 17 six-turn chains. They lay a playable core more reliably than they find hidden defects.
Coding agents can already turn a natural-language brief into a browser game from an empty workspace. That is still not a product. Logic, art, controls, UI, and playability have to survive together in one executable. Most existing game-agent benchmarks score the final artifact, or a single stage such as generation or repair. A clean build and a plausible screenshot do not prove that the controls work, that later edits keep the core loop intact, or that the agent can find defects it was not handed.
Tencent’s Hunyuan team and Lightspeed Studios coded complete human–agent co-creation traces by the intent of each artifact-changing turn. Three stages recur: first-pass generation, diagnosis and repair, and multi-turn product optimization. GameXpert-Bench turns those stages into three tracks. Credit is given only when the running game shows the intended behavior.
GameGen covers 97 briefs across 11 genres, 44 of them 3D. The agent gets one request, no template, no assets, and no prescribed engine, and must emit a browser-native game in a single session. After all models generate, a shared rubric is built from cross-model event analysis and split into completeness (core events), richness (bonus events), visual quality, and player experience, equally weighted. Completeness and richness require static code inspection plus live interaction. Visual quality and experience are human-scored.
GameFix starts from 50 internal Gold Games. Each level is mutated with 19 to 27 reversible defects spanning seven dimensions and 61 subcategories. Every level is run twice: Explicit Issue, with a full checklist, and Self-Discovery, where only subjective symptoms are shown and objective bugs must be found. Grading uses Playwright in headless Chromium. A bug counts as fixed only if its Fail-to-Pass probe passes and associated Pass-to-Pass probes still pass. The headline metric, Strict, is the height of the survival curve on the 90–100 band. Cliff is the drop from checklist mode to self-discovery.
GameOpt replays 17 JavaScript games through six turns and 102 requests covering gameplay, level design, numeric balance, art, UI, and audio. The repo is never reset. Scoring uses 701 criteria (604 positive, 97 regression) with difficulty-leaning weights, then subtracts triggered regressions.
On GameGen, Claude-Opus-5 leads at 79.7 overall, with completeness 94.4 and richness 72.0. Across 15 models, mean completeness is 77.5 and mean richness 46.1. The 3D subset sits 5.8 points below 2D, driven by completeness (−8.8) rather than visuals (−2.4). UI overlap or misalignment shows up in 15.2% of 1,455 runs. Of 43,081 event checks, 5.32% are implemented in code yet Fail or Partial at runtime; 56.0% of those are load or crash failures.
| Model | Overall | Completeness | Richness |
| Claude-Opus-5 | 79.7 | 94.4 | 72.0 |
| Claude-Fable-5 | 75.8 | 89.2 | 62.8 |
| Kimi-K3 | 71.3 | 85.2 | 56.0 |
| Seed-2.1-pro | 48.7 | 58.4 | 30.8 |
On GameFix, even Claude Opus 5 reaches only Strict 39.0; the median model sits near 14. Checklist mode compresses 17 models into about 13 points. Self-discovery stretches that span to about 38. Opus 5’s Cliff is 7.6; Hy3’s is 32.8. Named-bug localization is close to saturated. The remaining gap is whether the agent will play, verify, and clear roughly twenty independent defects on its own.
On GameOpt, Claude-Opus-5 scores 93.96 against Seed-2.1-Pro at 35.89. Across 612 measured runs, 42.6% are fully integrated, 49.5% playable but incomplete, and 7.8% unplayable. Same first-turn UI request: Kimi-K3 passes 39/39; MiniMax-M3 drops a closing style tag, the page stays blank, and all 39 checks fail.
For anyone shipping a coding agent as a game co-developer, this split is the useful one. Generation rankings reward a playable skeleton. The ranking moves once richness, 3D, layout stability, and runtime wiring enter. Repair rankings collapse if every bug is named, then reopen when the agent has to discover them. Optimization shows that a locally correct edit is cheap compared with keeping the whole product reachable.
A demo that “builds a game from one sentence” will look strong on GameGen completeness. The three tracks together match how people actually work with these agents: make something playable, find what is broken, then take product notes without killing the loop.
The Gold Games are closed, on purpose, to block memorization. Outsiders cannot reproduce those 50 levels. Packing 19 to 27 bugs into one session is an evaluation choice; the paper says real games rarely present that many at once. GameOpt confounds turn index with dimension: audio is always turn six, balance clusters at turn three, so the late-turn rebound is not evidence of long-context stability. All three tracks stay in browser JavaScript. Engine-native and client-native work is untested. GameGen’s shared rubric is pooled from model outputs, so it may overweight events models already emit and underweight designer-valued content that none of them produced.