Game-Guided Skill Discovery through Self-Play for Playable Agent Control
Seungeun Rho, Jeonghwan Kim, Xue Bin Peng, Sehoon Ha
cs.LG, cs.AI, cs.RO
2026-10-01
Agents self-play 1v1 games to learn 5-6 discrete skills; humans drive robots with those skills and solve unseen Maze/CubePush tasks at 84-100% success without retraining.
Unsupervised skill discovery tries to learn a reusable repertoire of motor behaviors without task rewards. The dominant objectives, mutual information (DIAYN, DADS) or Wasserstein dependency measures (METRA), push different skill codes toward distinguishable state distributions. Distinguishable is not the same as meaningful: on high-DoF agents, skills can differ in trivial ways that are easy to discriminate but useless to a person. Existing fixes add guidance, but language-guided methods need LLM descriptions across the whole state space, and reference-motion methods need costly datasets that also cap what can be learned. GGSD wants skills that are semantically distinct, interpretable, and directly drivable by a human.
GGSD grounds skill discovery in self-play over simple 1v1 competitive games: AntSumo, AntFencing, FrankaAirHockey, and G1Boxing. A game rule says what counts as winning and stays silent on how. A hierarchical policy is trained jointly:
The design choices are deliberate. Opponents come from a pool of past checkpoints rather than a mirror of the current policy; the latest is drawn with probability p in [0.6, 0.8] and the rest of the probability spreads uniformly over older ones, giving a piecewise-stationary opponent distribution in the spirit of fictitious play. The low-level policy and the discriminator see only proprioceptive observations, so skill semantics stay consistent no matter who the opponent is; that consistency is what makes the interface playable. Categorical entropy regularization keeps the high level from collapsing onto a few skills. On the engineering side, 4,096 parallel environments split into 10 groups that share frozen opponents to fit in GPU memory.
The most interesting emergent property is combos. Skill semantics are state-dependent: individual Franka skills just park the end effector over a region of the table, yet transitions between skills produce fast, purposeful strikes. In G1Boxing, skill 5 to 2 yields a backward lean for evading punches, while the reverse converts stored momentum into a heavy forward punch. Five or six buttons support far more behaviors than primitives, much like button combos in fighting games.
Self-play quality: after 70k updates, the final policy keeps win-and-draw rates above 50% against every earlier checkpoint; in G1Boxing it exceeds an 80% win rate against all past checkpoints except itself. Four human players each played five games of AntSumo against the final agent and won only two, a 10% human win rate.
Semantic diversity, measured by having Gemini-3.6-Flash caption each skill's rollout video and computing pairwise embedding distances: GGSD scores highest on all three embodiments, roughly 2x the strongest baseline (DIAYN, DADS, METRA) on Ant and G1, and 28% higher on Franka.
Human play on unseen tasks, with the low-level policy frozen and the high level replaced by keyboard input (trained in Isaac Lab, evaluated in MuJoCo, and featuring walls and cubes absent from the training games):
| Task | Success | Avg time |
| AntMaze | 100% | 82.7 s |
| AntCubePush | 100% | 62.2 s |
| G1Maze | 84% | 22.0 s |
Five participants, two of them authors, five trials each, with at most five minutes of practice.
Appendix B offers a clean decomposition: mutual information assigns behaviors to skill codes but does not make those behaviors meaningful; meaning has to come from an external source of structure. A game rule is about the cheapest such source available, a few lines of code with no motion dataset and no LLM labeling. For robot control, the practical draw is the interface: training produces a five- or six-button controller a human can operate immediately, and it transferred zero-shot to unseen tasks and a different simulator. Compared with reference-motion pipelines, this trades curated data collection for game design.
The authors state the central one: the skill repertoire is bounded by what the game demands, so a boxing policy will not learn dexterous manipulation. Appendix A concedes that state-dependent semantics have a limit; too many regimes and the same button means too many things for a person to track.
Other gaps show up on a close read. "A few lines of code" undersells the reward shaping documented in the appendix, with approach, facing, joint-limit, and self-contact terms per game. The human study has five participants, two of them authors, so the 84% G1Maze figure rests on a small sample. Baselines were compared only on the VLM diversity metric, never on human playability, so the playability advantage is argued indirectly. The diversity metric itself depends on one captioning model and could shift with another. And the precondition for composability, that the state space contains only a few behaviorally distinct regimes, is an observation here, not a training-time guarantee.