From Gameplay to Policy: Towards Scalable Robot Data Collection via Gamified Robot-Free Interaction
Zheng Li, Liang Zhu, Junzhe Wang, Huayuan Chen, Ziyun Liu, Jiahang Cao, Xinyu Sheng, Pei Qu, Yufei Jia, Ximeng Zhang, Jiarui Xie, Zizhao Yuan, Haoang Li, Yi Cai, Jinni Zhou, Jun Ma
cs.RO
2026-09-16
HKUST-GZ collects robot-free manipulation in a VR cooking game, pretrains affordance cues, and lifts few-shot success by 10.0 points in simulation and 18.3 on a real G1.
Generalist manipulation policies need large, diverse interaction data. Real teleoperation is slow, embodiment-specific, and hard to crowdsource. Scripted simulation is homogeneous. Handheld interfaces such as UMI still need custom hardware. In-the-wild egocentric video has no action labels and needs heavy filtering.
Games keep people engaged and let them invent their own solutions. Existing "gamified" robot collection mostly paints a score overlay on teleoperation, so a physical robot is still in the loop. This paper turns collection into a robot-free VR game and asks whether the resulting manipulation experience can help a policy when only a handful of real demonstrations are available.
Two stages.
Project Kitchen is a Unity cooking game. The Meta XR SDK tracks 26 joints per hand. Hand poses, object poses, grasp events, and task progress are logged at 30 Hz. Cuisine unlocking, a branching narrative, bilingual prompts, and live feedback keep players in the loop. Each recipe subtask has a success predicate on spatial, count, or volume state; when it fires, the stream is segmented and labeled automatically. Pouring uses fill fraction times vessel capacity, with no fluid simulation. Clips are then grouped into three reusable primitives: pick-and-place, pouring, and tool use (ladling).
Game2Policy does not imitate game actions. The contact point is the palm world position at grasp onset, projected into earlier frames with that frame's camera, and supervised as a Gaussian heatmap because a bowl rim has more than one valid grasp. The sub-goal is a relative pose in the contact frame, which cancels world and camera motion and is not treated as an executable robot target. Grasp intervals come from hand kinematics: per-recording finger-curl percentiles (35th/55th), dropping spans shorter than 5 frames, longer than 3 s, or with under 6 cm of pre-contact motion.
The affordance network freezes DINOv2, predicts a 64×64 contact heatmap with convolutions, and regresses the sub-goal from a pooled MLP. Pretraining adds lighting and background randomization. At deployment the head is queried online. Top-K heatmap peaks go into the policy: extra tokens for transformer policies (ACT, SmolVLA), FiLM concatenation for Diffusion Policy. The affordance head is jointly fine-tuned at a lower learning rate with behavior cloning.
Ten hours of gameplay are collected first. Simulation uses three LIBERO-Goal tasks, 5-demo and 50-demo regimes, three backbones, three seeds, and 50 rollouts per seed.
| Backbone | 5-demo target-only | 5-demo Ours | 50-demo target-only | 50-demo Ours |
| ACT | 65.5 | 76.4 | 75.6 | 79.1 |
| Diffusion Policy | 62.7 | 70.9 | 77.3 | 79.6 |
| SmolVLA | 66.1 | 76.9 | 76.5 | 83.5 |
The 5-demo mean gain is 10.0 points, already close to the 50-demo target-only mean of 76.5. A Scratch control that keeps the affordance branch but randomly initializes it matches target-only, so the lift is the game prior, not extra capacity. Ablations: dropping pretraining costs 10.1 points; dropping the contact point costs 7.7; dropping the sub-goal costs 3.1. K=1 costs only 2.1 on the mean, but Bowl-on-Plate's standard deviation jumps from 2.6 to 7.1.
The real robot is a Unitree G1 with a Dex1-1 parallel gripper, 10 teleop demos and 20 trials per primitive: pick-and-place 10/20 to 14/20, pour 8/20 to 12/20, tool use 12/20 to 15/20, 30/60 to 41/60 overall, a gain of 18.3 points. Scratch is 28/60.
Against EPIC-KITCHENS-100, signal dropouts fall from 0.48 to 0.05 per second and manual labels from 89,977 to 0. Against LIBERO scripted demos, approach spread is about 2.3–2.6× and contact-chain length is 17.7 versus 1.04 and 0.98. A 10-person user study reports less upper-limb fatigue, lower cognitive load, and higher felt diversity than teleoperation.
Crowdsourcing robot data usually assumes a robot in the room. This work decouples collection from embodiment, takes ground truth from the game engine, and bridges the gap with embodiment-invariant affordances. For a lab that can afford only a few real demos, ten hours of VR play is a cheap prior. It does not replace real data. It turns few-shot learning from fitting a policy from scratch into fine-tuning a head that already points at contacts.
The authors list three: collection scale is still small, supervision stops at affordance rather than actions, and the visual game-to-real gap remains. They plan broader deployment, 3D Gaussian Splatting for appearance, and retargeting hand motion onto dexterous hands. As of this paper there are 10 hours, one cooking game, a parallel gripper, and 10 real demos per task, so crowdsourcing is a claim about the interface, not a completed campaign. The sub-goal is explicitly a human-hand-frame conditioning signal, not an executable trajectory. Hand-to-gripper transfer is tested; dexterous hands are not. The user study has 10 people, enough to illustrate, not to settle.