Embodied First Exam: 1,156 tasks on 47 robots test coding agents, best agent solves only 34% in standard setting
burny_tech · x · 2026-11-12
A multi-university team introduces Embodied First Exam, the first robotics benchmark built for coding agents — testing whether LLM agents can control robots via direct control, generated code, or self-trained policies rather than visuomotor policy learning.
Scale: 1,156 tasks across 35 suites and 47 robots, including 9 new suites covering spatial puzzles, dexterous manipulation, multi-robot interaction, dynamic whole-body control, and sim-to-sim engineering.
Three settings: Standard (sensor observations only, no resets — closest to the real world), Realtime (environment keeps running while the agent reasons), and Privileged (full simulator state with unlimited resets, serving as an upper bound).
Results: Among 8 agents, the strongest — GPT-6 Astra in Codex — achieves only 34% success on the 456 core tasks in the Standard Setting (77% with full simulator access, 30% under realistic constraints on matched tasks); in Realtime evaluations, 63% of tasks remain unsolved by any agent. The paper labels failure stages to compare agents, settings, and categories.
More from Embodied
- LeRobot explains why scaling in the wild matters for physical AI — micoolcho · 2026-11-12
- RoboCap-10k: 10,000 hours of open egocentric robot data across 21 industries released — micoolcho · 2026-11-12
- Apple affirms Vision Pro commitment; glasses rumored to be in secret development — Scobleizer · 2026-11-12
- ContactBench 1.0: A Benchmark for Contact-Rich Robot Manipulation With Printable Assets — leto__jean · 2026-11-12
- AMD runs MolmoAct2 robot policy locally on a Ryzen AI Max+ 395 mini PC — DJiafei · 2026-11-12
- Open-Source CoRL Paper Browser: 2,100+ Robot Learning Papers, a Decade of Searchable Research — promptdom · 2026-11-12