Same model, two placements: agentic control completes 18/20 LEGO tasks vs 6/20 for code-as-policy
paigeinsf · x · 2026-09-24
SE3 Labs ran a controlled comparison of where a robot's intelligence should live, using the same OpenAI GPT6-Astra model, the same bimanual YAM station, and the same LEGO pick-and-place task on a real-world remote eval stack:
- Code-as-policy: Astra writes and tests a Python policy, then the code runs frozen with no further model calls — 6/20 placements completed
- Agentic control: a fresh Astra session reads camera snapshots and issues commands through a control harness every episode — 18/20 completed
The gap wasn't grasping but recovery: code-as-policy lifted the brick in 14/20 episodes yet failed the rest without retrying; the agent needed multiple grasp attempts in 9 of its 18 completions, including one episode where it dropped the brick, reacquired it, and placed it at 419s. The tradeoff is speed: median release time 175s for agentic vs 131s for code-as-policy among successful runs, with the agent also granted up to 600s to recover vs 180s. The report includes a 0–100 progress rubric and a 100-point quality score (grasp 30 / transport 30 / placement 40), scored on-site by a human operator. Full report and all 40 episode videos are public.
More from Embodied
- First 1-Bit Multimodal Model Runs Locally on AI Smart Glasses via Snapdragon AR1 — IgorCarron · 2026-09-24
- Dev builds autonomous 2D village with Jev, eyes real-time robot decision-making next — claud_fuen · 2026-09-24
- Robot Fashion Is Here: Custom Outfits for Quadrupeds After ICSR 2026 Show — heatherknight · 2026-09-24
- After Astra: where does value go when frontier labs ship robotics brains? — mihdalal · 2026-09-24
- Robotics' GPT-3 moment may just be GPT-6 itself, argues investor — mihdalal · 2026-09-24
- Qualcomm and Prism ML Run 1-bit Bonsai VLM Locally on Snapdragon AR1 Smart Glasses — Scobleizer · 2026-09-24