Embodied First Exam benchmarks coding agents on 47 robots across 1,156 tasks; best agent hits just 34%
MLL-Lab · hf · 2026-11-12
- Researchers from Northwestern, CMU, UC Berkeley, Stanford and others (incl. Fei-Fei Li, Yuke Zhu) introduce Embodied First Exam, the first agent-native robotics benchmark letting coding agents (Codex, Claude Code, etc.) choose their control approach: direct control, code-based control, or training a policy.
- Scale: 47 robots, 35 suites, 1,156 tasks, 2,288 tests, with 9 new suites covering spatial puzzles, dexterous manipulation, multi-robot interaction, dynamic whole-body control and sim-to-sim engineering; robots range from single-arm/bimanual to humanoids, quadrupeds, drones and musculoskeletal systems.
- Settings: Standard (sensor observations only, no environment resets; a Realtime variant keeps the env running while the agent reasons) and Privileged (full simulator state, unlimited resets, serving as an approximate upper bound).
- Results: Of eight agents tested, the strongest — GPT-6 Astra in Codex — reaches only a 34% success rate on 456 core tasks under the Standard Setting; the paper also labels failure stages to compare failure modes across agents.
More from Embodied
- Abbeel lab's SuperMimic replaces physics sim with differentiable world model, ~1000x faster sim-to-real — HirokatuKataoka · 2026-11-12
- Berkeley open-sources RoboSimStudio: AI agents program industrial robots without training data — berkeley_ai · 2026-11-12
- NXP ships pre-tested drone platform: initial flight in 30 minutes out of the box — shashib · 2026-11-12
- Open-source dual-arm robot solves tangram puzzles with 90 demos, ~60% success — DanGrover · 2026-11-12
- Robotics has three regional playbooks: China volume, Europe inspection, US pilots — ingliguori · 2026-11-12
- Model Autonomously Designs a Surgical-Robot-Style Robotic Arm CAD After 75 Minutes — _Stocko_ · 2026-11-12