ZimaBlue: Evolving Generalizable World Action Models through Scalable Video Pre-training
Xionghao Wu, Yijun Yang, Shiyang Zhou, Haoze Sun, Jianhui Liu, Songsong Yu, Jiyao Zhang, Wenbo Li, Bo Wang, Guoqing Ma, Lin Song, Renjie Liao, Shenghe Zheng, Wei Tang, Xiaojuan Qi, Yanwei Li, Yuan Zhang, Zhuotao Tian, Haoyang Huang, Nan Duan
cs.CV
2026-09-01
ZimaBlue pretrains a 5B WAM on 120k hours of egocentric video, lifting Franka zero-shot success from 36.1% to 77.8% and latency to 33 ms on RTX 4090.
Generalist manipulation is short on scalable physical experience, not on policy architectures. Action-labeled robot trajectories are expensive and visually narrow. Egocentric video is cheap and diverse, but it has no executable actions. Off-the-shelf video generators are trained for plausible clips, often with bidirectional context and long chunks, which does not match a robot that must act on observations that have already happened.
ZimaBlue treats this as a World Action Model (WAM) problem: learn causal visual dynamics from action-free video, ground those dynamics in cross-embodiment trajectories, then calibrate on the target robot. The work comes from Joy Future Academy. The backbone starts from Wan2.2-TI2V-5B.
Training follows a three-stage pyramid. Stage I is causal video pretraining on sources such as EPIC-KITCHENS, EgoDex, DROID, and AgiBot, scaled past 120,000 hours. Actions are placeholders and the Fast branch is off. Stage II jointly flow-matches video and action on DROID, AgiBot, Galaxea, and RoboMIND2-Franka, mapping native interfaces into a shared 100-dimensional semantic layout (end-effector, gripper, joints, torso, base, dexterous hands) with validity masks. Stage III specializes Slow on the target embodiment, freezes it, and trains a 0.5B Fast tower.
Slow is a 5B DiT that predicts future video latents plus auxiliary actions. Fast consumes the latest observation and cross-attends to video K/V caches from the first 12 Slow layers. At deployment the two streams are asynchronous: Slow updates world guidance at low frequency, Fast keeps action continuity with an RTC-style prefix. DMD distillation (8 DiT evaluations down to 2 per branch) plus compile cuts the closed-loop from 450 ms to 33 ms, about 30 Hz on an RTX 4090.
Real-robot evaluation uses a 7-DoF Franka on 12 zero-shot tasks (8 Standard, 4 Perturbed), 10 trials each. Every variant shares the same DROID post-training.
| Config | Standard | Perturbed | Overall |
| target robot only | 46.7% | 15.0% | 36.1% |
| +6k h multi-embodiment | 57.9% | 22.5% | 46.1% |
| +60k h video | 82.9% | 35.0% | 66.9% |
| +120k h video | 87.9% | 57.5% | 77.8% |
| π0.5 | 65.4% | 32.5% | 54.4% |
| DreamZero | 61.7% | 37.5% | 53.6% |
Cross-embodiment data mainly lifts contact-rich execution (microwave closing 0/10 to 9/10). Scaling video from 60k to 120k hours adds 5.0 points on Standard and 22.5 points on Perturbed. Slow-Fast versus Slow-only moves Standard from 80.8% to 87.9% and Perturbed from 30.0% to 57.5%. In simulation, LIBERO-Plus is 86.7% zero-shot and 92.0% after task SFT; RoboTwin 2.0 is 94.7%/94.3% Clean/Randomized; RoboCasa365 Composite-Unseen is 16.5% versus 7.9% for ABot-M0.6, still behind Xiaomi-Robotics-1, which uses 100,000 hours of real-robot data. Simulation numbers use the Slow branch only.
This is a controlled answer to whether video can be the main scaling axis for robots: the same DROID post-training, video from 0 to 120k hours, more than doubles zero-shot success, and the extra points land on unseen visual shifts. WAMs are usually too slow for contact; Slow-Fast brings that cost down to 30 Hz on a consumer GPU. The practical recipe is video pretraining plus cross-embodiment alignment. Target-robot data is still required, just less of it.
The full model still scores 5/10 to 7/10 on each Perturbed task. Remaining failures are stalled intermediate states, lost contact, and grabbing the tablecloth. The three simulation suites never turn Fast on, so the 30 Hz system is not compared there. Ten trials per real-robot task is noisy; the paper correctly weights suite means. 120k hours is still small against internet video, and the authors treat it as a scaling axis to keep growing.