SG-WAM: Enhancing World-Action Models with VLM-Based Semantic Guidance
zhenjun_zhao · x · 2026-08-13
Existing World-Action Models (WAMs) rely primarily on visual cues rather than language instructions, causing predicted videos to be semantically misaligned with instructions and degrading action prediction accuracy. This paper proposes SG-WAM to overcome this limitation.
- VLM Planner: Introduces a vision-language model (VLM) as a semantic planner to enhance the instruction-grounding capacity of WAMs.
- Semantic Foresight: Trains the planner to predict text-grounded and spatial-aware semantic foresight. Text-grounded foresight identifies correct target objects, while spatial-aware foresight provides scene geometry for precise manipulation.
- Injection: Injects this foresight into the WAM as high-level semantic guidance, ensuring both future-video generation and action prediction faithfully follow language instructions. Superiority is demonstrated in both simulation and real-world experiments.
More from Embodied
- AutoAgent0: VLM-powered agentic runtime lifts long-tail driving success by 28.3pp — yining_hong · 2026-10-16
- CarryOn: dual-loop harness adds personalization and safe recovery to frozen VLA robots — Zixuan Wang · 2026-10-16
- FoveaVLA replaces wrist cameras with learned foveation, hitting 70.8% success — Haoyu_Xiong_ · 2026-10-16
- Musk teases two-way Grok x Optimus integration: Grok in the bot, bot in Grok — beffjezos · 2026-10-16
- SimVTA uses simulator-rendered video to post-train driving world action models with RL — OpenDriveLab · 2026-10-16
- Robotics data startups compared: Scale AI tops $1B revenue as 9 LLM vs robotics data suppliers benchmarked — FinanceYF5 · 2026-10-16