Coding Agents Take a Robotics Exam: Best Model Solves 34% of Core Tasks, 77% with Simulator Access

Embodied First Exam

Kangrui Wang, Qineng Wang, Changhe Chen, Pingyue Zhang, Chi Gui, Feishi Wang, Canyu Chen, Wenli Xiao, Xiaoyu Huang, Tonghe Zhang, Linjie Li, Zihan Wang, Minyeong Kim, Yiyou Sun, Mingkai Deng, Jinyu Hou, Qi Zhu, Han Liu, Eric P. Xing, Huaxiu Yao, Zhiwen Fan, Yuejiang Liu, Dawn Song, Pieter Abbeel, Li Fei-Fei, Jiajun Wu, Yuke Zhu, Guanya Shi, Ruohan Zhang, Manling Li

cs.RO, cs.AI

2026-11-09

A new benchmark puts coding agents in control of 47 robots; the best model solves 34% of hard tasks with sensors only, 77% with full simulator access.

What problem this solves

Coding agents like Codex and Claude Code already fix issues in real repositories, and recent work has pointed them at robots: driving the robot directly, writing its control program, or training a policy in simulation. Nobody had measured this route systematically. Robotics benchmarks are built for visuomotor policy learning with fixed interfaces, and coding-agent benchmarks like SWE-bench and Terminal-Bench impose no physical constraints. Which robots can a general-purpose coding agent actually run, how much does it lose without simulator access, and where does it fail were all open questions. The Embodied First Exam (EFE), from a team spanning eight institutions including Northwestern, answers them with 1,156 tasks in 35 suites across 47 robots of nine body types, from single arms and dexterous hands to humanoids, quadrupeds, and drones, and runs eight mainstream agents through it.

Method

EFE constrains the result, not the method: the agent picks its own control approach, and the framework fixes the starting state, the resources, and the grading. Three approaches are supported: direct control (the model emits action chunks), code to control (it writes a planner or controller), and train a policy (it trains in simulation and deploys); mixing is allowed. The infrastructure reuses Harbor from Terminal-Bench, so one command runs any model-harness combination, while the verifier sits in an isolated container the agent cannot see, with the success-rule code hidden to block reward hacking.

Nine new suites (411 tasks) fill gaps in existing benchmarks: untying knots and assembling IKEA furniture (RoboARC), dexterous hands turning pages and opening jars (DexHandExam), catching and striking moving objects (RoboDynamicExam), humanoid athletics and full matches (Embodied Olympics), goals given only as an image (iSpatialGym), hazardous requests mixed with benign ones (Embodied Safety Exam), and controllers tuned on a nominal simulator then deployed to hidden dynamics (RoboEngineer). The other 26 suites (745 tasks) adapt existing benchmarks, with expert policies and demonstration data removed so nothing can be copied.

The central design runs each task in up to three settings that share the scene, success rule, and time limit, so results pair up cleanly. Standard mimics a real robot: the agent sees only the robot's own sensors (joint states, RGB-D), cannot reset by default, and the simulation pauses while it thinks. Realtime keeps the world running while the agent reasons. Privileged grants full simulator state plus unlimited resets, so the agent can brute-force an oracle solution; its success rate serves as an approximate upper bound. Attempts get 30 minutes to 4 hours (median 1 hour) with compute caps. Headline numbers use a screened core of 456 hard tasks (901 tests). For failure analysis, 427 tasks are split into stages (open the fridge, tomato inside, fridge closed again), and a separate labeling agent locates the stage where each failed attempt went wrong.

Results

AgentStandard (456)Privileged (391)Realtime (54)
GPT-6 Astra (Codex)33.6%77.5%31.5%
Claude Opus 5.5 (Claude Code)25.0%59.3%7.4%
Gemini 4 Argon (Antigravity CLI)24.1%42.7%9.3%
Kimi 3.1 Pro (Claude Code)10.1%37.6%7.4%
GLM 5.3 Flash (Claude Code)2.4%9.7%0%

The gradient is steep: the top three more than double the other five and together solve only 48% of Standard tests, yet their solved sets diverge, with Claude Opus 5.5 and Gemini 4 Argon between them solving 64 Standard tests that GPT-6 Astra fails. Realtime is hardest, with 63% of its tests unsolved by any agent, against 50% of the Standard tests on the same 54 tasks.

The paper decomposes the Privileged-to-Standard drop cleanly. On the 391 paired tasks, GPT-6 Astra falls from 77% in Privileged to 28% on the Standard interface; adding back exact object state raises it to 45%, unlimited resets to 53%, both to 68%. Resets are worth more than state. Precise stages (tight fits, target poses, contact with moving objects) rise 24 to 46 percentage points in failure rate, coarse stages only 1 to 20; assembly goes from 34% to 80%. At precise stages, resets recover about 16 of 91 failures, exact state only about 2.

Failures concentrate. For the top three agents, 56% of failed-attempt pairs on a test fail at the same stage (chance: 31%), usually the task's first stage (65%). Once reached, material stages such as wiping fail 66% of the time, body stages such as walking 45%, and mechanism stages like opening a door 20%. Weaker agents die earlier: they fail without making the required contact at 9.1% to 18.5% of contact stages, against 2.7% to 3.7% for the top three.

The control approach follows the setting: 61% of sampled Standard attempts steer the robot directly, while 82% of Privileged and 97% of Realtime attempts write a program. The soccer match is the cleanest experiment: given a GPU and an RL library, 10 of 12 Standard attempts trained a policy on the spot, and every win came from a trained policy. GPT-6 Astra won 10-8 with the GPU and lost 3-13 on CPUs with a hand-written gait, its robots falling 59 times to the opponent's 7. Across all 1,489 sampled attempts, only 6 trained a policy.

Why it matters

This is the first systematic measurement of the coding-agent-as-robot-controller route, on infrastructure anyone can run: one command, an isolated verifier, public trajectories on HuggingFace. Three findings carry over directly. The privileged-to-deployed gap is over 40 points, so privileged results should not be read as deployment projections. For precise manipulation, retry resets beat exact state perception, a concrete hint for what a real-robot interface should preserve. And a harness swap moves the same model by up to 12 points (GPT-6 Astra: 34% in Codex, 22% in Claude Code), so evaluations and model selection must treat the harness as a variable. The Standard interface deliberately mirrors real hardware, and the paper demonstrates it on a physical robot arm playing buzz wire, with systematic real-robot evaluation as the stated next step.

Limitations

The paper names three hard ones: one attempt per test, so no confidence intervals and part of the between-agent gaps may be run-to-run variance; stage and approach labels come from LLM agents, and the main labeler (GPT-6.1 Sol) shares a family with GPT-6 Astra, a bias risk; everything runs in simulation. The core set was screened with GPT-6 Luna, same family again, which may bias it toward tasks that family fails. Realtime covers only 54 dynamic tasks. Cross-provider harness runs go through an adapter, and Antigravity CLI kills the session when Gemini's content filter blocks a reply, which may understate Gemini. Stage-category failure rates mix suite and robot composition, so they are not causal. The 34% headline also sits on a screened hard core, so it is not comparable to absolute numbers from other benchmarks.

Terms

Source

What people are saying

Related papers

All paper explainers