CMU paper: one unmodified computer-use agent wins prizes across four embodied AI contests
wellecks · x · 2026-11-10
CMU's Aggarwal and Welleck argue embodied tasks can be framed as interaction with digital software: render the physical environment on a screen and act via standard keyboard/mouse.
Key findings:
- A single unmodified Computer-Use Agent (CUA) solves complex embodied tasks (3D navigation, segmentation) purely through keyboard/mouse inputs
- With one CUA design and only software wrappers changed, their entries won 1st, 2nd, 3rd, and 6th place across four ECCV workshop competitions (OpenSUN3D, MeiVs2-Text, DialNav, VLNVerse)
- A standard 9B VLM emitting text coordinates nearly matches specialized VLA π0.5 in robotic manipulation sim (56% vs 59% success)
- GUI vs text API interaction gap is negligible for frontier models, making GUIs a universally viable bridge
- They push back on the trend of specialized VLA architectures with dedicated action heads
Roadmap: visual grounding via simulation data, RL in software-wrapped environments, and inference-speed optimization for real-time interaction. They predict embodied AI will converge into the same general-purpose model that codes, does math, and uses desktops.
More from Embodied
- SimEX: agents bootstrap robot skills in simulation, then fold towels in the real world — chris_j_paxton · 2026-11-10
- Amazon Ring unveils $250 autonomous home security drone that patrols and streams live video — Polymarket · 2026-11-10
- NVIDIA-led study: unblocking VLM gradients in VLA training lifts manipulation progress from 10.3% to 32.9% — stepjamUK · 2026-11-10
- Google DeepMind launches Gemini Robotics 2.5 with 96% accuracy sorting Mercedes-Benz parts — GoogleDeepMind · 2026-11-10
- SlashVibe is a physical desk controller for managing multiple running AI coding agents — Scobleizer · 2026-11-10
- Path Robotics raises $300M Series E at $2.45B valuation to expand physical AI — Rewkang · 2026-11-10