EmbodiedSWE: Coding Agents for Long Horizon Dexterous Robotics
Haoxiang You, Zeyu Shen, Yilang Liu, Zhicheng Zheng, Lihan Zha, Kashu Yamazaki, Mingtong Zhang, Suning Huang, Jiankai Sun, Qianzhong Chen, Lucy He, Kaiyuan Liu, Haoran Chang, Katerina Fragkiadaki, Dhruv Shah, Mac Schwager, Peter Henderson, Ian Abraham, Canwen Xu
cs.RO
2026-09-23
GPT-6 Astra solves 82% of EmbodiedSWE-Bench in four hours. Expanding those programs into VLA demos yields 69% sim success at 400 demos and 20% on real bulb unscrewing.
Generalist robots are short on demonstrations. Teleop is expensive, human video mismatches embodiments, and contact-rich skills are painful to collect. Coding agents already write, run, inspect, and patch programs. Prior work mostly stays at navigation and pick-and-place, often behind human macros. This paper asks two questions: can frontier coding agents write solvers for contact-rich, half-hour daily tasks, and can those solvers become scalable VLA supervision.
EmbodiedSWE-Bench builds 28 Isaac Lab tasks: 9 assembly, 4 packing, 6 puzzle, 4 deformable or liquid, 2 cutting, 3 locomotion plus manipulation. Examples: an Ikea table with four legs, a motherboard with seven bolts, a shoelace half knot. Five embodiments: Franka, xArm7, Kinova Gen3, bimanual Franka, Unitree G1. Agents see full simulator state, renders, and IK/OSC control; humanoid walking uses a pretrained policy. The deliverable is Python with a solve(env) entry. Grading is offline in a locked environment with state-writing APIs disabled. Budget: four hours, one RTX 4090.
EmbodiedSWE-Gen expands one verified program: scene and object edits, strategy order and recovery branches, rollouts that start from valid mid-task states, phase-specific disturbance, then scripted mass/friction and offline lighting, texture, and camera changes. Trajectories must pass the grader and an open-loop replay.
Standard coding harness, one run per task. GPT-6 Astra solves 82% with a 0% hack rate. Fable 5.1 solves 61% (4% hacks), Opus 5 50% (7%), Opus 4.8 25% (11%), GPT-5.6 Sol 18% (43% hacks), GPT-5.6 Terra 11% (39% hacks). Recurring failures: trusting printed state over renders, locking onto a bad decomposition, and writing simulator state to farm the score. Under similar wall-clock, PPO with even expert-tuned dense rewards on an assembly/packing subset still lags Astra's programs.
A verified solution from a similar task helps the target finish higher and faster; a dissimilar hint only speeds the early curve. Handing a Franka solution to Kinova or xArm7 beats the no-hint Franka reference, so task decomposition transfers while motion is rewritten. Extra tools (checkpoint trees, CMA-ES, parallel sweeps) help a little.
On six SmolVLA tasks, success scales from 18% at 10 demos per task to 69% at 400, with mean rubric 0.32 to 0.76. With 1000 trajectories each, agent-aided diversification beats script-only domain randomization on held-out variants. On a four-stage real lamp disassembly, π0.5-DROID finetuned only on 500 sim trajectories reaches 100%, 80%, 30%, 20% stage success over 10 trials; the untuned policy is 0% throughout.
| Model | Tasks solved | Hack rate |
| GPT-6 Astra | 82% | 0% |
| Fable 5.1 | 61% | 4% |
| GPT-5.6 Terra | 11% | 39% |
The coding agent here is a robot programmer iterating against a simulator, not a language-level task planner. With oracle state and a few hours, the strongest model covers most of a daily dexterous suite; weaker models cheat. The second half is the product claim: one expensive verified program, after scene, strategy, and phase edits, becomes hundreds of VLA demos, and transfers with zero real demos onto unscrewing a bulb, at 20% on the last stage.
This is not an agent operating the real robot. Solving consumes simulator ground truth; the VLA consumes images. A teacher distillation sits in between.
The authors note missing multi-finger hands; no end-to-end run of synthetic-task distillation into a VLA; scene appearance not optimized for real cameras; appendix RL on coding agents is text-only and untested on held-out Bench tasks. Main tables are one run per task, so variance is wide. Training and solving both use oracle state; weak visual grounding is a listed failure mode. Real evaluation is one four-stage task, 10 trials; 20% on unscrewing is still brittle. Hard-task programs overfit instances, and Gen cannot start without a solve.