GameHorizon Suite: Multi-Horizon Data and Evaluation in Gameplay
Yiran Wang, Xingyilang Yin, Junfu Pu, Guangzhi Wang, Kaifeng Li, Mingyu Ouyang, Huiqiang Sun, Lingen Li, Cheng Cheng, Wangbo Yu, Honghao Chen, Xiaodong Cun, Chi-Man Pun, Zhiguo Cao, Ying Shan
cs.CV, cs.AI
2026-09-22
GameHorizon records 5,000 hours of AAA play with three-horizon labels. Across 47 models, current-action beats future planning; most stay at or below 10% online.
Gameplay evaluation has lacked one yardstick that jointly scores perception, planning, and keystrokes. Existing corpora sit in Minecraft or mini-games, skip language, or rely on a handful of online rollouts that are noisy and hard to reproduce. Dedicated game agents drop planning to keep control frequency high. General VLMs can plan, yet they have not been scored in a shared action space.
Tencent ARC Lab ships a three-part suite: an automatic annotator, 5,000 hours of AAA play, and an offline-plus-stepwise-online benchmark covering short operations, medium goals, and long strategies.
One hundred skilled players recorded 2K/60fps video locked to keyboard-mouse events: 4,571 clips and 411.03 million actions. After dropping cutscenes and alt-tabs, 4,341 hours (86.8%) enter instruction labeling.
Annotation is bottom-up. Key action transitions cut short clips; a VLM writes 1–5 second L1 operations. Adjacent clips merge by continuity into 1–2 minute L2 goals and 5–8 minute L3 strategies, with dynamic programming enforcing those ranges. The corpus holds 6.184 million distinct instructions: about 5.95 million L1, 189k L2, and 47k L3. One distinct L1 appears every 2.63 seconds, and every frame aligns to all three layers.
The offline track turns aligned triplets into four-way MCQs: T1 single-horizon action, T2 goal decomposition, T3 cross-horizon consistency, 1,000 items each, plus ten diagnostic variants (2,000 items). The online track runs in Minecraft: 10 causal and 10 thematic long tasks, 62 subtasks. A failed subtask resets to its success state so later steps can still be scored.
Forty-seven models, more than one million calls. The 44 models that can answer questions average 64.7% offline, ranging from 44.6% to 80.2%. Difficulty is ordered: T1 57.3%, T2 65.1%, T3 71.6%. GPT-6-Astra leads at 80.2%, then Gemini 3.8 Flash at 77.3%. Kimi-K3 is the only open-weight model in Tier 1 at 74.5%. GUI agents land in Tier 4 (UI-TARS-1.5-7B 53.0%). Dedicated game agents sit near chance on unseen AAA titles: NitroGen 22.0%, JARVIS-VLA 28.1%, though the same JARVIS-VLA reaches 54.2% in-domain on Minecraft.
| Setting | Current action | Future action |
| Frames only | 44.5% | 34.6% |
| short instruction | 58.6% (+14.1) | 34.8% |
| multi-horizon | 60.0% | 41.8% (+7.0) |
Bottom-up abstraction hits 98.4%; top-down decomposition only 69.5%. Online ranks track offline tiers. GPT-6-Astra completes 45.0% of long-horizon tasks; 10 of 12 tested models stay at or below 10%.
This is the first public corpus that is large-scale, AAA-heavy, human-action, and densely multi-horizon at once. Offline MCQs are reproducible. Stepwise resets separate “does not know the recipe” from “knows it and still misplaces the item.” For game VLA training, short instructions mainly help current actions; medium and long instructions help future planning. A single coarse goal label is not enough.
Dedicated game agents collapse outside their training title. GUI post-training does not transfer to AAA play. Future-action planning is still the weak joint.
The online track lives in Minecraft because most AAA titles hide internal state, so it does not cover the 21 recorded games. Labels come from a VLM without large human audit. Game agents are scored on T1 only via sequence alignment, a different protocol from the QA models. MCQs test recognition, not closed-loop high-frequency control. Chain-of-thought drops small models by 5.9–8.2 points. Even the leader reaches only 45% on long-horizon online tasks.