REAL embodied agent handles vague commands, hits 78.3% success on real robot
机器之心 · wechat · 2026-09-15
Researchers from Shanghai Jiao Tong University and Shanghai AI Lab introduced REAL, an open-world mobile manipulation framework accepted to ECCV 2026. It abandons two idealized assumptions—privileged perception and clear instructions—and unifies active exploration, visual grounding, intent disambiguation, and physical manipulation into one closed loop.
The system uses Qwen3-VL-8B-Instruct as its brain, with MCP as the unified tool interface (four-step exploration tools, Pick/Place control, and an Ask tool for clarifying vague user requests). Training is two-stage: SFT for tool alignment, then GSPO online RL for closed-loop exploration, error recovery, and proactive questioning. On the 241-task REAL-Bench, REAL-8B scores 56.9% on the simulator-user-loop task, beating zero-shot baselines including Gemini-3-Pro (53.8%) and GPT-5 (52.3%).
For sim-to-real, the high-level policy transfers zero-shot to an ARX LIFT2 dual-arm mobile robot, with a fine-tuned π0.5 handling low-level skills—achieving 78.3% end-to-end success over 60 real-world episodes and an 85.3% execution rate across 600 VLA primitives.
More from coding & agent
- Developers Arguing Over Coding Skills Is Like Taxi Drivers Vs. Waymo, Says Dev — CtrlAltDwayne · 2026-09-15
- One-line prompt makes an AI build "the cutest little game" with zero follow-up questions — paraschopra · 2026-09-15
- Dev validates idea on Reddit: dirt-cheap OpenAI-compatible API for open-source models — Extension-Pen-109 · 2026-09-15
- Plus users stretch GPT-6 Astra limits by pairing it with cheap flash models for execution — haider1 · 2026-09-15
- From Ollama to vLLM: a roadmap for scaling LLM deployment — kalyan_kpl · 2026-09-15
- Blender previz plus MiniMax H3 makes AI dance videos far more stable — Hailuo_AI · 2026-09-15