Embodied First Exam: 1,090 embodied tasks show agents crumble outside privileged settings

ManlingLi_ · x · 2026-11-18

Manling Li's team released Embodied First Exam (EFE): one line of code runs coding agents across 31 simulation suites, 44 robots, and 1,090 embodied tasks, covering spatial-constraint puzzles, multi-robot Olympic sports, real-time eval where the world won't pause, dynamic whole-body control, and sim2sim AutoResearch. Try it: harbor run -d embodied-first-exam -a <harness> -m <model>.

BiGym 2.0 is one of the 31 suites. EFE's three settings tell a stark story: under privileged access, Fable 5.2 and GPT-6 Astra solve all 10 core tasks; using only the robot's own cameras, the best score drops to 6; in realtime, at most 1 — sharply quantifying the privileged-to-real gap.

Related event: Embodied First Exam benchmark finds coding agents struggle in physical world(2 posts)→

Original post →

More from Embodied

Embodied channel →