EFE benchmark from 8 universities tests coding agents on 1,156 robot control tasks
机器之心 · wechat · 2026-11-22
Teams from Northwestern, CMU and 6 other universities release EmbodiedFirstExam (EFE), a unified benchmark evaluating coding agents controlling simulated robots across 1,156 tasks, 35 task suites and 47 robots.
Design
- Only outcomes are fixed—agents may directly control robots, write control code, or train policies
- Three modes per task (privileged/standard/real-time) progressively tighten perception, resets, and timing
- Harbor-based packaging, one command to run, independent scoring, resource caps
Findings on the 456-task core set
- Standard mode: best agent ClaudeFable5.2 solves half; weakest under 7%; 7 of 22 suites have ≥half tests unsolved by all five strong agents
- Fine-grained phases suffer far more from tightened conditions: assembly failure jumps 25%→76%, placement 18%→31%
- Controlled experiments show unlimited resets matter more than precise state (19%→36% vs no gain)
- Control strategy follows resources: 63% direct control in standard mode, 97% closed-loop programs in real-time; with GPU, 25/28 soccer tasks trained policies
- Framework matters: GPT-6Astra solves 34% in Codex vs 27% in ClaudeCode
Failure attribution: task graphs localize failures to stages; 25% of tests concentrate >80% of failures in one stage. On the double-catch task all 10 agents fail—ClaudeOpus5.5 estimated the catch window at 1.3–1.6s vs 1.4s to reposition the cup.
Limitations: single run per test; annotator GPT-6.1Sol is itself a tested agent from the same family as GPT-6Astra.
More from Embodied
- Tesla FSD v15 renders the real world in real time, claims 25x safety gain — xiaohu · 2026-11-22
- Robotics veterans clash: general models are the key to the last 6% problem — BradPorter_ · 2026-11-22
- Google DeepMind and YC startup Hebbian host world's first Robot Use Agents hackathon — DynamicWebPaige · 2026-11-22
- Developer has three AI models plan the same robotics project, then merges the best ideas — doodlestein · 2026-11-22
- Tesla FSD XV: wrong-asset training renders reportedly fixed — yunta_tsai · 2026-11-22
- Tesla camera shows surprisingly strong low-light signal, developer says — yunta_tsai · 2026-11-22