LLMs Begin Directly Controlling Robots
DorsaSadigh · x · 2026-07-15
The author notes spending the past year on LLM post-training, but is now witnessing a major leap: "LLMs are starting to directly control robots, collect data at scale, and automate numerous robotic processes."
The core insight comes from VIA: Visual Interface Agent for Robot Control. Researchers equipped agents like Claude Code and Codex with browser-based visual interfaces, allowing them to control robots using a mouse and keyboard via MCP tools, much like a human would—and it actually worked.
The key takeaway isn't just robot control; it's the integration of general-purpose agents, visual interfaces, and robotics, pointing toward the automation of future robot data collection and training pipelines.
More from Embodied
- Physical world agent swarm demo shared on X — pduan · 2026-10-26
- Looking for a wearable mic with best battery life that feeds recordings to personal agents via MCP — altryne · 2026-10-26
- 'Lamp Swarm': a desk of lamp-shaped robots shows what physical AI work looks like — dee_hw · 2026-10-26
- Tencent's HiSTEVE 1.0: hierarchical VLA agent plays Minecraft with native keyboard and mouse, hits 85% SOTA — _akhaliq · 2026-10-26
- SJTU's VaM lets robots learn online from mistakes, 95%+ success over 4-hour autonomous runs — chris_j_paxton · 2026-10-26
- DIY humanoid PhoneGirl day 4: collision tests, servo rework, 160° knee bend — _Stocko_ · 2026-10-26