WALL-SS: Scaling Long-horizon World Models via Next-Scale Autoregression
Maeve Zhang, Rain Sun, Xiang Wang, Cyril Zhang, Shalfun Li, Meng Cao, Howard Lu, Ethan Chen, Harry Jhou, KZ Zheng, Lights Shi, Regis Cheng, Lorenzin, Robert Wang, Victor Yao, Gody Li, Elise Mon, Yohann Tang, Ryan Yu, PS Zhang, Vincent Chen, Hang Su, Roy Gan, Hao Wang, Qian Wang
cs.RO
2026-08-27
WALL-SS matches real-robot policy success at r=0.93 over 600 pairs (MAE 0.062), holds 60s streaming error under 0.5% of the image diagonal, remaining optimistic at insertion.
Most robot world models still generate a clip. Actions are extra conditioning, and anything that has to live across clips (memory, streaming, reward updates) is bolted on afterwards. The mapping from a control to its consequence is not part of the generative factorization.
That becomes a real problem as soon as you want minute-long rollouts or closed-loop policy evaluation. Autoregression writes every generated frame back into context, so small errors compound across object identity, contact outcomes, and the action history that produced them. WALL-SS, from X Square Robot, puts action conditioning, bounded memory, and reward alignment inside one next-scale generative process.
The visual generator is initialized from InfinityStar. Each future observation is built coarse-to-fine: layout and contact first, residual detail later. Actions are not a global prompt. They are scale-aligned condition tokens. A query at a given scale sees only the control for that time window and camera; future controls and other views stay masked.
Three pieces share that hierarchy.
Training data is anchored on the full AgiBotWorld-Beta release: about 1.00 million trajectories, processed into 987,508 captioned clips from 165,560 source videos, plus private bimanual recordings and UMI handheld-gripper data. Intervention and recovery trajectories are kept as their own bucket: live takeovers, and rollback-and-replay branches from the state just before an error. The target failure mode is a magnet grasp, where closing the gripper makes the object jump into the fingers.
On the WorldArena-style embodied video benchmark (200 in-distribution and 100 OOD tasks, 80% with actions), the gap is largest on physically grounded metrics. Image quality still trails CogVideoX-5B at 0.718.
| Method | Traj. Acc. | Action Following | Interaction | Instr. Following | Image Quality |
| InfinityStar | 0.251 | — | 0.484 | 0.406 | 0.602 |
| Wan2.2-14B | 0.159 | — | 0.476 | 0.394 | 0.690 |
| Cosmos3-Nano | 0.202 | 0.044 | 0.516 | 0.410 | 0.693 |
| WALL-SS | 0.539 | 0.290 | 0.546 | 0.471 | 0.697 |
In a 60-second streaming pour, frame-wise trajectory error stays below 0.5% of the image diagonal through the contact-rich phase. Memory ablations (no dream forcing, recent-clip only, full-history KV) are shown as curves; the paper does not report a single scalar for those variants.
Reward alignment versus the supervised checkpoint: action following 0.264 to 0.290, trajectory accuracy 0.512 to 0.539, cross-clip boundary error 0.118 to 0.104. Appearance metrics stay within evaluation noise. The same action expert, reading the aligned visual state, raises real-robot Task Progress from 64.6 to 69.1. The rewards never see actions, progress, or success.
Closed-loop policy evaluation is the harder test. Five WALL-WM checkpoints run on six tasks with 20 matched initial states, 600 generated-vs-real pairs. Success-rate correlation is r=0.93. Across 30 task-checkpoint cells, MAE is 0.062, signed bias +0.028, calibration slope 0.84. Within-task pairwise ranking holds 89% of 59 untied pairs, mean Spearman 0.88, mean selection regret 0.025. Outcomes match in 527 of 600 episodes, balanced accuracy 0.88. Residual optimism sits at contact: insertion transition bias +0.12. Of 332 real failures, 45 still succeed in simulation (FPR 0.14).
On the tabletop bimanual robot, mean Task Progress is 69.1 against π0.5 at 49.6, DreamZero at 44.1, and LingBot-VA at 34.0. Press Button in Order scores 63 versus a best baseline of 31; Classify Items as Shape scores 80 versus 55.
The product people want is not prettier robot video. It is a simulator that can rank policy checkpoints. Correlation 0.93 and pairwise ranking 89% put WALL-SS in "good enough to shortlist checkpoints" territory. Contact and insertion remain optimistic, so weak checkpoints get inflated.
Because next-scale autoregression exposes token likelihoods, the visual generator can be aligned directly. Teams already using video world models for policy evaluation get a recipe: scale-aligned actions, scale-compressed memory, rewards that score dynamics fidelity only.
The closed-loop numbers are on in-house WALL-WM checkpoints and an in-house tabletop bimanual. Transfer to other embodiments and policy families is unproven.
There is no Limitations section. The discussion stays at the slogan of moving from appearance generation to reusable interactive understanding. The experiments are more specific.
Contact and insertion optimism is systematic. Calibration slope 0.84 lifts weaker checkpoints. Each cell has N=20; the paper says not to over-read cell-level differences.
The evaluation surface is in-house. The video benchmark is built from their mixture, the closed-loop policies are WALL-WM, the robot is their tabletop bimanual. Baselines such as π0.5 share task definitions, observations, and scoring rubrics; whether they share training data is not stated. Parameter count is missing from the text; the qualitative figure labels the backbone Infinity-8B. Private and intervention data volumes are not quantified. Sixty seconds is minute-scale, not shift-scale. Alignment starts from a strong supervised checkpoint and is KL-tethered, so the gains are residual by design.