Genie: Generative Interactive Environments
Jake Bruce, Michael Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, Yusuf Aytar, Sarah Bechtle, Feryal Behbahani, Stephanie Chan, Nicolas Heess, Lucy Gonzalez, Simon Osindero, Sherjil Ozair, Scott Reed, Jingwei Zhang, Konrad Zolna, Jeff Clune, Nando de Freitas, Satinder Singh, Tim Rocktäschel
cs.LG, cs.AI, cs.CV
2024-02-23
DeepMind's Genie learns latent actions from unlabeled internet video to build an 11B per-frame-controllable interactive environment, with actions that transfer to new tasks.
Building an interactive, controllable world model used to require either action-labeled data (available in games, hard in the real world) or settling for video-level control with no per-frame agency. The internet has vast unlabeled video, almost none of it with action labels. Genie learns a generative interactive environment from that unlabeled video, unsupervised: give it an image, some text, or even a sketch, and it produces a playable virtual world.
The key idea is the latent action: the model infers a discrete action code between adjacent frames itself, with no ground-truth action labels. Three components make up the model. A video tokenizer (ST-ViViT) uses a VQ-VAE to turn frames into discrete tokens with interleaved spatiotemporal attention, scaling linearly in the number of frames. A latent action model (LAM) uses only 8 codes and a VQ-VAE objective to infer actions from history and the next frame; the small vocabulary is deliberate, to keep things controllable and human-playable. A dynamics model uses MaskGIT for next-frame prediction and treats actions as additive embeddings. The final model is about 11B parameters. A recurring engineering choice is the ST-Transformer: spatial layers attend within a frame, temporal layers attend across frames at the same position, which saves memory.
On the in-house Platformers dataset, the 11B model reaches FVD 40.1 with pixel-input LAM versus 38.8 for token-input LAM; a tokenizer ablation gives ST-ViViT 81.4, spatial-only ViT 114.5, and C-ViViT 272.7.
| Metric | Setting | Value |
| Δt-PSNR (controllability) | Genie, Platformers | 1.91 |
| Δt-PSNR (controllability) | token-input baseline | 1.33 |
| Δt-PSNR (controllability) | Genie, RT1 robotics | 2.07 |
| FVD | curation 55M to 6.8M videos | 61.4 to 54.8 |
In behavioral cloning, an agent trained on Genie's latent actions solves CoinRun at nearly 100% with only 200 expert samples, matching the oracle that uses ground-truth actions, while a random agent sits near 0%. Training used 256 TPUv5p chips and about 6.6e22 FLOPs.
This is the first demonstration that video alone, with no action labels, suffices to learn a per-frame-controllable interactive environment, and that the learned latent actions transfer to unseen tasks through behavioral cloning. That is a foundational result for world models, embodied intelligence, and procedural content generation, and Genie 2 and 3 followed the same path. The latent-action recipe also caught on in robotics, and the "no real labels" angle flagged in the discussion is exactly its selling point.
It inherits the usual autoregressive failure mode and hallucinates implausible futures. Memory is capped at 16 frames, which strains long-horizon consistency. Inference runs at about 1 FPS, far from real-time interaction. The main results are compute-heavy, and the authors provide only a smaller reproducible case study. The FVD and other metrics are measured on in-house datasets, so there is no unified head-to-head against contemporary world models.