Training Agents Inside of Scalable World Models
Danijar Hafner, Wilson Yan, Timothy Lillicrap
cs.AI, cs.LG, cs.RO, stat.ML
2025-09-29
Dreamer 4 is the first agent to obtain Minecraft diamonds purely offline, with 0.7% diamond and 29% iron-pickaxe success, using about 100× less data than VPT's web-labeled set.
World models should let an agent rehearse inside its own simulator, skipping unsafe or slow real interaction. Earlier Dreamer agents are fast and accurate in narrow environments but lack capacity for messy object physics. Controllable video models such as Genie 3 generate diverse scenes, yet they miss precise mechanics and often need many GPUs to roll a single scene in real time.
Getting a diamond in Minecraft is a 20,000-plus-step mouse-and-keyboard task; experienced humans average about 20 minutes. VPT labeled 270K hours of YouTube from 2.5K hours of contractor play, then added 194K hours of online RL to reach diamonds. Many deployments cannot put a half-trained policy into the real world. This paper tightens the goal: diamonds from offline data only, raw pixels and low-level controls.
Dreamer 4 trains in three phases. First a causal tokenizer (masked autoencoding, MSE+LPIPS) and a dynamics model. Then task tokens are inserted and behavior plus reward are learned with length-8 multi-token prediction. Finally the transformer stays frozen and only the policy and value heads update on imagined rollouts with PMPO.
Dynamics uses shortcut forcing: diffusion forcing's per-timestep noise plus shortcut models' step-size conditioning, so each frame needs K=4 forwards. The network predicts clean latents (x-prediction) rather than velocity, which cuts high-frequency error that would accumulate in long rollouts. A ramp weight on signal level spends capacity on noisier-but-informative steps. Past frames are lightly noised (τ=0.1) so the model tolerates its own imperfections.
The backbone is a block-causal transformer with separate spatial attention and temporal attention every four layers, plus GQA to shrink the KV cache. Tokenizer is about 400M parameters, dynamics about 1.6B. Minecraft training uses the 2541-hour VPT contractor set at 360p and 20 FPS. Keyboard is 23 binary heads; mouse is 121-way foveated discretization. Behavioral cloning is applied only on task-relevant clips; the dynamics loss stays on uniform clips to avoid optimistic imagination.
Evaluation is 60-minute episodes in random worlds from an empty inventory, 1000 episodes. VPT (finetuned) typically reaches sticks at about 53% success. A Gemma 3 VLA clone reaches iron pickaxes 11% of the time. Dreamer 4 is above 90% through the stone pickaxe, 29% for iron pickaxe, and 0.7% for diamonds. Imagination RL helps more on harder milestones and also shortens time-to-item. World-model features beat Gemma 3's general visual features for cloning.
A human playing 16 interaction tasks inside the world model finishes 14/16 with Dreamer 4, 5/16 with Oasis large, and 0/16 with Lucid-v1. Context is 9.6 s, about 6× prior Minecraft world models, at 21 FPS on one H100, matching the game's 20 FPS.
Action labels can be scarce. With actions on only 100 of 2541 video hours, action-conditioned generation reaches 85% of full-action PSNR and 100% of SSIM; 10 hours yield 53% / 75%. Action conditioning trained only in the Overworld still extrapolates to Nether and End videos that never had actions. A design cascade drops FVD from 306 to 57 and lifts inference from 0.8 FPS to about 21 FPS.
This is the clearest recipe so far that offline world models plus imagination RL can handle long-horizon pixel control. For robotics, where online trial-and-error is expensive and unsafe, the split is useful: most knowledge can come from unlabeled video, a little aligned action data grounds the body, and policy improvement can stay inside the model.
A 0.7% diamond rate is small in absolute terms. Its meaning is that every offline baseline in this setting is 0. Stone-pickaxe 90% and iron-pickaxe 29% say the early chain is already usable; diamonds remain a tail event.
The world model is not a game clone. Memory is about 9.6 seconds, and inventory items often smear or swap. Two of sixteen interaction tasks still fail. 0.7% diamonds is far from a 20-minute human run, and the paper does not compare against online Dreamer 3 or VPT-RL under matched action spaces. Imagination RL optimizes a learned reward model; if that model is biased, the policy farms phantom progress. Training used 256–1024 TPU-v5p chips. The robotics section shows counterfactual video, not a policy on hardware.