Berkeley and DeepMind Propose Instruct-to-Act: Making World-Model Controllers Language-Instructable

berkeley_ai · x · 2026-09-22

Zineng Tang et al. (UC Berkeley, Vector Institute, Google DeepMind) present Instruct-to-Act at COLM 2026, a system that decouples planning and control for language-instructable agents.

Key ideas:

• VLMs are strong at high-level reasoning but struggle to produce reliable, low-latency action sequences, while world-model controllers excel at fast observation-to-action control but lack open-ended task guidance. Instruct-to-Act conditions a learned world-model controller on sparse, high-latency language instructions from a VLM.
• Controllers are made language-instructable without manual labels: segments of policy rollouts are relabeled with instructions via a post-hoc annotation method, then optimized with behavior cloning.
• The controller autoregressively generates high-frequency actions, can follow instructions from any VLM or human, and flexibly swaps between VLMs.
• It extends naturally to multi-agent settings where VLMs communicate through trained controllers as actuators, without multi-agent RL.

The paper reports results across embodied environments, scaling trends for larger controllers and VLMs, and ablations on instruction cadence, planning frequency, and online vs. offline planning latency. Code and paper are available.

Original post →

More from Embodied

• Saudi brand Ceer unveils Exobot, an AI-designed concept humanoid — aziz4ai · 2026-09-22
• Weekly AI Recap: Helix 2.5 Lifts Household Robot Success to 56%, World Models Meet Robot Control — TheTuringPost · 2026-09-22
• SemiAnalysis tears down A20 on TSMC N2, DRAM revealed in iPhone 18 Pro Max package — dylan522p · 2026-09-22
• Redditor predicts Navier-Stokes will be solved before an AI robot can autonomously clean your bedroom — Crazyscientist1024 · 2026-09-22
• TI's Edge AI Play: Decoupled TinyEngine NPU Cuts Inference Latency Up to 90x on MCUs — 智东西 · 2026-09-22
• BCI-Sonics raises $28M Pre-A for non-invasive ultrasound brain-computer interface with AI closed loop — 机器之心 · 2026-09-22

Embodied channel →