Think Like a World Model, Act Like a VLA: Distilling World-Model Representations into Compact Robot Policies
Trung Dao, Sankalp Yamsani, Jaden Park, Joohyung Kim, Yong Jae Lee
cs.RO, cs.CV
2026-09-21
A cosine term aligns a 0.8B VLA to cached world-model features: 97.9% on LIBERO (from 95.3%) and 50.5% on RoboCasa-GR1, still 32 ms and 1.86 GB on an RTX 5090.
VLAs learn to emit actions from observations with no pressure to model how the world pushes back. Robustness then tracks demo coverage, and viewpoint, lighting, or clutter breaks them. World models do the reverse: predicting the future forces temporal and causal structure, and transfer is better, but rolling the future is slow. DreamZero needs about 3 s and 45.9 GB on an H100. π0.5 runs in 65 ms on a consumer RTX 5090. A world model does not fit in the control loop.
This paper splits what a world model knows from generating the future. Physical scene structure already lives in internal features. Generation is only the objective that produced them. A student can match those features without dreaming.
The student is QwenGR00T: a Qwen3-VL backbone plus a GR00T-style flow-matching action expert. The main loss remains velocity regression on demonstration actions; teacher actions are unused. An extra term mean-pools image tokens per camera, maps them with a two-layer MLP, and applies cosine alignment to a cached teacher feature, λ=0.5.
The default teacher is the Qwen3-VL-8B reasoner inside Cosmos 3's understanding tower, layer 24, 4096-d per view. It runs once over training frames into a memory-mapped cache. Training never loads teacher weights. The projector is dropped after training. Inference is one backbone prefill and four flow steps, identical to the undistilled student.
LIBERO, four suites, 50 episodes per task, averaged over two seeds and two GPUs. Distilled 0.8B: Spatial 99.3, Object 99.2, Goal 99.4, LIBERO-10 93.8, mean 97.9±0.5. The undistilled control in the teacher ablation is 95.3. Published VLAs under 4B mostly sit between 78.7% and 95.3%.
RoboCasa-GR1, 24 GR1 humanoid environments trained jointly: 48.2±2.1 to 50.5±2.3, 4.3 points behind the 4B sibling at 54.8%. Real robot, 30 trials:
| Task | 4B π-style | 0.8B undistilled | 0.8B distilled |
| Single-arm fruit | 93.3 | 83.3 | 93.3 |
| Single-arm egg | 70.0 | 46.7 | 60.0 |
| Bimanual handover | 53.3 | 40.0 | 46.7 |
Failures cluster in the last centimetres: early release, closing on the rim, eggs slipping. Recognition works; contact does not. The gain holds across student scale, backbone, alignment layer, and teacher family (Cosmos / Fast-WAM / V-JEPA2-AC), which score 97.9, 96.9, and 96.5 on LIBERO.
World-model grounding does not require rolling video in the loop. One offline cache and one cosine term leave latency and memory equal to the small VLA: 32 ms and 1.86 GB on an RTX 5090. A 0.8B policy sits near much larger LIBERO numbers. This is a cheap representational prior, not a new architecture. Gains are consistent and small, about 2–3 points in simulation. Contact physics barely moves, as the egg task shows.
The lift is modest, and RoboCasa already moves several points across GPUs, so 2.3 points sits near evaluation noise. There is no dedicated limitations section; the contact failures are the authors' own evidence that a visual prior does not fix force control. Building the teacher cache still costs a one-off compute pass. Alignment is on pooled image features, not actions or future frames, so how much temporal structure transfers is indirect. LIBERO is near ceiling, and a 2.6-point gap is a weak proxy for real robustness.