MIT's Phillip Isola on 'robot-use agents': coding alone won't crack dexterity
Xingyu2017 · x · 2026-09-09
Phillip Isola published a short blog arguing that AI agents like Claude controlling robots marks an important change in robotics' trajectory. A follow-up discussion notes the open question: computer-use agents already accomplish tasks via generated code instead of mouse and keyboard actions—how far can coding alone go for robots?
Key points:
- LLM agents will keep improving at reasoning and adapting to new tasks, but dexterous manipulation and fast feedback are hard via code alone—akin to autonomous driving's long tail.
- One path: build and run simulations during reasoning, then search for solutions—viable in predictable settings with limited real-time feedback needs.
- Another: learn from real-world interaction data (egocentric, UMI, teleops), either inside a foundation model or via specialized policies behind well-designed interfaces.
Related event: MIT's Isola Proposes "Robot-Use Agents" as AI Controls Robots(4 posts)→
More from Embodied
- π0.5 study: raising demos from 10 to 438 lifts Place Ball in Bowl success from 7% to 83% — stingning · 2026-10-27
- GPT-6 Astra controls dexterous hands zero-shot at just 5.2% success in 500 trials — stingning · 2026-10-27
- HandsOn benchmark: 66 verified tasks and 18,000 human demos for dexterous manipulation — stingning · 2026-10-27
- DexPort: a handheld interface with a real robot hand for dexterous-hand data collection — pabbeel · 2026-10-27
- ROS names Alejandro Hernández Cordero new project lead, eyes 2027 release — ___Mufasaa · 2026-10-27
- ROS Meetup coming to Google's NYC office Oct 29 with speakers from Tangent, Cohesive Robotics, Intrinsic — kscottz · 2026-10-27