EmbodiedFirstExam: 1,156-task benchmark shows top coding agents solve only half
新智元 · wechat · 2026-11-19
Researchers from Northwestern, CMU and six other universities released EmbodiedFirstExam (EFE), a unified benchmark wiring coding agents into simulated robots: 1,156 tasks, 2,288 tests, 47 robots across 23 simulators, open-sourced with code and trajectories.
Key results
- On the 456-task standard-mode core set, the best agent ClaudeFable5.2 (ClaudeCode) solved only 50%, GPT-6Astra 34%, Kimi3.1Pro 10%; 169 tasks stumped every agent, and no single winner — four strong agents collectively solved 60 tasks the leader missed.
- Three escalating modes: privileged (full sim state, unlimited resets) 80% → standard (sensors only) 47%, showing the gap to real-robot conditions; realtime mode left 46% of tests unsolved.
- Real-robot demo: agents steered a physical arm through a wire-loop game; GPT-6Astra averaged 94% progress over its last four runs and twice completed the full course.
Analysis findings
- Ablations on GPT-6Astra show restoring unlimited resets (53%) beats restoring exact state (45%) — "seeing precisely ≠ acting precisely," especially for fine alignment failures.
- Failures cluster: different agents often fail at the same stage (55% vs 30% random); material-contact stages fail 56% of the time once reached, body stages 39%, mechanism stages 17%.
- In soccer tasks with GPU+RL libraries provided, 25/28 attempts trained policies, and all wins came from trained policies; with CPU only, agents hand-wrote gaits and mostly lost.
Limitations: single runs per test, some labels produced by a tested model, mostly simulation-only; the team plans repeats and systematic real-robot evaluation.
More from Embodied
- SE3 Labs Releases First Open YAM Real-Robot Teleop Dataset: 500 Bimanual Episodes — paigeinsf · 2026-11-19
- 1.5 years under review: researchers urge releasing AI papers immediately after submission — carlosdponx · 2026-11-19
- Purple metallic robot dog spotted walking indoors: 'Greetings from the future' — technollama · 2026-11-19
- Sound-based robot destroys tumors without cuts: histotripsy treats 2,000+ patients — TinfoilTricorn · 2026-11-19
- Behind the scenes of a robot dog field expedition at the Matterhorn — ki_ki_ki1 · 2026-11-19
- Reddit user's AI device Dot proactively followed up on his ear condition next morning — HannsGruber · 2026-11-19