CodeActionBench: Evaluating Agentic Code-as-Policy for Embodied Manipulation
Yiheng Lyu, Xueying Jiang, Wenhao Li, Shijian Lu, Gongjie Zhang
cs.RO, cs.AI
2026-09-28
CodeActionBench has models write dual-arm policies from RGB, with no depth or object poses. Over 675 attempts span 2.7-73.3%; GPT-6 Astra with Codex CLI solves 22 of 25 tasks.
Naming a mug is not a grasp. The model still has to pick a contact, turn it into a pose the arm can reach, and change the plan when the outcome is not what the code expected.
Two current routes hand that middle stretch to something else. A vision-language-action model (VLA) learns an observation-to-action map from robot demonstrations. OpenVLA was trained on 970,000 real manipulation trajectories. Code-as-Policy has a model write programs over a robot API at inference time, but published systems often still receive object poses, grasp proposals, or skills from specialist modules, and sometimes from simulator state. The success rate then does not say how much spatial judgment the model did itself.
CodeActionBench, from Nanyang Technological University, takes those aids away. It has 25 manipulation tasks in SAPIEN on RoboTwin 2.0's ALOHA-AgileX dual-arm robot: grasping, transport, handover, tools, arrangement, stacking, brief contacts, and articulated objects. No task fine-tuning and no demonstrations. Depth, object state, specialist perception or grasp modules, and pre-written task policies are all outside the interface.
What gets scored is a model plus its harness, the wrapper that builds requests, checks tool calls, and manages context. All nine configurations share the tasks, the robot API, the budgets, and the verifier. Seven use one reference harness. Claude Opus 5 is also run under Claude Code, and GPT-6 Astra is run only under Codex CLI. Once the harness changes, the score belongs to the whole agent.
The model sees RGB plus camera intrinsics and extrinsics. It does not see depth or object poses. Metric 3D is its job: triangulate a point across two wrist frames after a small motion, use the known gripper size as a ruler, or supply an object-size prior. It can also probe and intersect a camera ray with a plane built from contact plus an assumed normal. Geometry tools only compute. They do not pick pixels, and they do not certify the assumption. The model sets the grasp direction. cuRobo checks reachability and plans the joints. Its collision body is the robot alone, so a plan that passes can still hit the table or the objects.
Tools can be called one by one or composed as branching Python through runcode. One program counts as one tool call. Motions inside it still spend simulated time. After an interruption the achieved state comes back, and the model chooses the recovery.
Twenty-one tasks are scored on the physical state at the end, and the condition has to still hold. Hammer strike, bell press, stapler press, and bin pour are scored on events during the episode. done stores the model's own success claim. The verifier never returns its verdict. Domain randomization is off, and each task uses one fixed scene seed for every configuration. The tool-call budget is about 14 times the expert time in minutes, and simulated time is at least 60 seconds. Reference scripts written with privileged state finish all 25 tasks inside those budgets. The scripts are withheld. Three attempts per task make 675 episodes.
184 of 675 attempts succeed, 27.3% overall. No configuration finishes all 25 tasks.
| Configuration | Attempt success | Solved at least once in three tries |
| GPT-6 Astra (Codex CLI) | 73.3% (55/75) | 22/25 |
| Claude Opus 5 (reference harness) | 49.3% (37/75) | 19/25 |
| Claude Opus 5 (Claude Code) | 45.3% (34/75) | 14/25 |
| Gemini 3.6 Flash (reference) | 20.0% (15/75) | 8/25 |
| GPT-5.6 Sol (reference) | 16.0% (12/75) | 9/25 |
| Qwen 3.8 Max (reference) | 14.7% (11/75) | 6/25 |
| Grok 4.6 (reference) | 12.0% (9/75) | 6/25 |
| Kimi K3 (reference) | 12.0% (9/75) | 5/25 |
| Claude Sonnet 5 (reference) | 2.7% (2/75) | 2/25 |
On the reference harness, Opus 5 at 49.3% leads Gemini 3.6 Flash at 20.0%. Astra never ran that harness. Against Opus on it, Astra reaches 73.3% versus 49.3%, with 15.1% lower total inference cost and 60.0% shorter median wall time. The median, 477.2 seconds, is the shortest of the nine. The figure marks Astra's 75 attempts at $252.87. Gemini's total is $52.00, the lowest of the nine.
The two Opus harnesses differ by 4.0 percentage points in success and by five tasks in coverage, 19 versus 14. Astra solves 15 tasks on every attempt and misses three tasks on every attempt. Only Astra lifts the pot, and it does so 3/3. Only Opus on the reference harness empties the bin or hangs the mug, each at 1/3.
Fifty-five subgoal checkpoints and 72 spatial checkpoints, averaged with equal weight per task, split arrival from completion. Astra covers 87.3% of the spatial checkpoints and 76.0% of the subgoals. The two Opus runs are nearly tied on space, 73.6% and 73.4%, and farther apart on subgoals, 60.6% and 55.8%. By charged call 22, Astra's spatial coverage already beats every other configuration's final value. Entering the region does not finish the task.
Of 491 failures, 280 never confirm arrival, 77 never confirm object control, and 122 arrive but miss the subgoal or lose it later. Twelve of Astra's 20 failures sit in those later two bins. Stronger systems stall less often on finding the region and more often on holding, alignment, and keeping state.
Astra ends 72/75 attempts with done, and 68/75 claims match the verifier (90.7%). On failed attempts it correctly reports failure 70.0% of the time (14/20) and overclaims success on 15.0%. The two Opus systems correctly report failure on only 36.8% and 29.3% of their failures. Sonnet 5 leaves 70/75 attempts without an explicit claim, 69 of them because a budget ran out.
The four strongest configurations send 85.6% to 92.6% of charged calls through programs. The other five send 15.6% to 35.2%. Astra's program error rate is 0.46% (10/2167), and it triangulates on all 75 attempts. Both Opus runs probe contact much more often (71/75 and 67/75). The paper does not ablate the tools.
With the interface held fixed, the gap on one harness is readable: 49.3% against 20.0%. Astra with Codex CLI moves the ceiling to 73.3% and 22 of 25 tasks, with no fine-tuning and no demonstrations. Planning and low-level control stay in the shared API. The model owes the targets and the revision after feedback.
Seventy-five attempts cost $252.87, with a median wall time of 477.2 seconds. Multi-turn code edits are not cheap. The paper states that the latency is a poor fit for time-limited contact. A 20% success rate is not ready to run.
A final pass-fail score mixes three endings: never arrived, arrived but failed to keep the relation, and claimed success on a failure. Subgoals, failure stage, and the done claim have to be read together.
The stated scope is 25 simulated tasks, one dual-arm body, and one fixed scene each. Three attempts measure variation on that same instance. New scenes, new bodies, and real robots are untested. Multi-turn delay is also a poor fit for contact that cannot wait.
73.3% is Astra plus Codex CLI. It is not a bare-model ranking of GPT-6 against Opus. On the shared harness the leader is Opus 5 at 49.3%. Moving that same Opus to Claude Code drops success by 4.0 percentage points and coverage by five tasks. Grok 4.6 is capped at 8,192 output tokens per request, and Qwen runs at Medium reasoning effort while the others run at High. Those differences still share one chart.
The collision model leaves out the table and the objects, and the paper does not count how many failures are plans that were cleared and then collided. No VLA is scored on these 25 tasks, so 73.3% does not locate code-writing relative to a trained policy. Spatial checkpoints borrow successful expert motions and are looser than the verifier. Each task gets only three attempts, and provider APIs can change under a rerun.