ST-WAM: Semantic-Temporal World Action Model for Robust Manipulation under Visual Distribution Shifts
Mingxin Wang, Bin Hu, Bin Qian, Kaitao Jiang, Haoning Wu, Feng Yan, Bowen Jing, Ruiyang Hao, Enyi Wang, Kangning Niu, Yandan Yang, Mu Xu, Yan Wang, Houde Liu, Tianlun Li
cs.RO, cs.CV
2026-07-31
Pixel-future world models hallucinate under visual shift. ST-WAM switches to DINOv3 features and history retrieval, beating Fast-WAM by 21.3 points zero-shot and doubling real-world success.
World Action Models (WAMs) have the robot learn actions and predict future visual frames jointly, an appealing idea. But they rely on pixel-level future-frame prediction for supervision, which tangles task-relevant state change with task-irrelevant visual content. Under visual distribution shift (changed background texture, lighting), the model falls apart.
The authors name the failure Training-Distribution Hallucination: when predicting futures from visually shifted observations, the predicted frames drift back toward training-domain appearance rather than staying faithful to the new scene. They manually audit 180 predictions and 70.6% show it.
The cause is representation. A controlled study shows DINOv3 features stay more stable for the same state under visual shift (cosine similarity 0.904), while Wan-VAE latents manage only 0.686, and DINOv3 still separates distinct task states. DINOv3 is the more shift-robust and still discriminating semantic representation.
ST-WAM (Semantic-Temporal WAM) makes a deliberate choice: instead of correcting the contaminated future prediction, switch to DINOv3 as the shared semantic representation and keep the fine-grained VAE dynamics.
Two modules do the work. The first is Dual-Space Future Experts (DSFE), a three-branch Mixture-of-Transformers: a visual future expert (5B Wan2.2 video DiT) predicts future VAE latents, a semantic future expert (1B DiT) predicts future DINOv3 features, and an action expert (1B DiT) produces actions. A carefully designed attention mask controls information flow: current VAE and DINO tokens can attend to each other but cannot read future or action tokens; the two noisy future streams attend to each other for mutual refinement; action tokens can read current streams but neither future stream. The action must be decided from the present.
The second is Current-Anchored Intent Retrieval (CAIR), which pulls evidence from history. Qwen3-VL turns the current observation and instruction into semantic tokens that query the DINOv3 history of the last 4 frames (t-24, t-16, t-8, t-1); two cross-attention layers retrieve intent tokens injected only into the action expert.
It trains end-to-end with no extra embodied pretraining or task-level annotation, and at inference it never has to actually generate the future frames.
On LIBERO, ST-WAM averages 98.7%, already tight against baselines (Fast-WAM 97.6%, π0 94.4%). The real gap is robustness to shift.
The clearest evidence is zero-shot LIBERO-Plus (with visual perturbations):
| Method | Zero-shot accuracy |
| π0 | 53.6% |
| Fast-WAM | 51.5% |
| OpenVLA-OFT | 69.6% |
| ST-WAM | 72.8% |
ST-WAM beats Fast-WAM by 21.3 points. Broken down by perturbation, camera shift is +39.0 points and noise is +41.8, so the gains land exactly where old methods fail.
Real-world numbers are starker: nominal 79.3% for ST-WAM versus 64.7% for Fast-WAM; under visual shift 61.5% for ST-WAM against only 25.8% for Fast-WAM, which more than halves. From nominal to shifted, Fast-WAM drops 38.9 points while ST-WAM drops 17.8. On RoboTwin 2.0, ST-WAM reaches 92.77%, also slightly above Fast-WAM (91.83%) and LaWAM (91.22%).
Deploying robot policies in the real world is blocked in large part by fragility: a camera jitter or lighting change and the policy fails. This paper pins the blame squarely on pixel-level future prediction and answers with a DINOv3 semantic representation plus history retrieval that needs no extra pretraining yet sharply improves shift robustness, doubling success under visual shift. For teams building general manipulation policies it is direct engineering reference.
What it resists is visual distribution shift; changes in physical dynamics (object weight, friction) and transfer across embodiments are listed only as future work and are not tested. The cost is 1.24x inference latency (756ms versus 609ms), a real concern for high-frequency control. The paper has no dedicated limitations section; these limitations are judgments drawn from reading it.