Embodied Playground: A Long-Horizon Agentic Robotics Benchmark for Physical, Strategic, and Spatial Reasoning
Feng Chen, Pei Zhou, Lixuan He, Yunchao Yao, Li Sun, Shenghua Gao, Mingyu Ding, Kaifeng Zhang, Kaichen Zhou, Yi Ma
cs.RO
2026-10-20
HKU and collaborators' Embodied Playground scores long-horizon robot agents on 410 tasks; on EP-100, Opus-5.5-extra hits 35.85% suite-balanced Pass@2 and GPT-6-Astra 19.94%.
Robot benchmarks usually test a control policy: after the object or the table changes, does the grasp still land? Put a foundation model in the loop and the question moves. The model has to read the scene, pick a next step, turn it into control code, and rewrite the rest of the plan from what execution actually did. Tip a part, miss a shot, or spend the last projectile, and later actions that used to be legal disappear.
That cross-stage dependence is long-horizon reasoning. Tower of Hanoi must stay on legal intermediate states, with shortest solutions of 1, 3, 9, and 31 moves. A Kongming lock, a burr puzzle of interlocking pieces, has to satisfy assembly order and geometry at the same time. Launcher scenes add finite ammunition, a required hit order, and protected objects. A plan can be legal and still fail in contact and dynamics.
Meta-World, RLBench, CALVIN, and LIBERO measure policy learning and generalization. BEHAVIOR-1K and RoboCasa stretch household work and still score whether the task finishes. Embodied Playground scores a full episode of a foundation-model agent: track progress, meet intermediate constraints, revise from feedback. An independent checker splits the final goal, the task rules, and physical validity. The group is at HKU, Sharpa, Tsinghua, Stanford, UNC Chapel Hill, and MIT.
The collection holds 410 entries. Each entry is a task configuration paired with a robot body. Forty Hanoi boards on three embodiments account for 120 of them. Launchers contribute 100, on a dual-arm Panda in MuJoCo. Rubik's cubes contribute 60, Kongming locks 60, pocket cubes 20, and bookshelves 20, all in Isaac Lab. ManiSkill adds 30 household, geometry, and dynamics tasks. Bodies cover parallel grippers and dexterous hands. Dexterous-hand items admitted to EP-100 are mostly easier native configurations.
The agent writes Python through a public SDK, runs it, and edits it, rather than emitting a joint trajectory in one shot. The calls are describe, observe, step, and finish. An observation read does not advance simulated time. Assistance stacks in four levels. L0 is RGB, proprioception, calibration, and public parameters, with code limited to the standard library, NumPy, Pillow, and algorithms the agent writes itself. L1 adds reconstruction, inverse kinematics, motion planning, and symbolic solvers. L2 allows pretrained grasp and contact models. L3 adds read-only simulator state such as pose, velocity, and contact. Weights stay frozen. Reference trajectories and off-the-shelf controllers are withheld.
EP-100 keeps 100 of the 410 so a full evaluation stays affordable. Five suites, 16 types: Launch 34, Tower 21, Lock 16, Cube 12, Home 17. Launch moves from aiming to causal chains, breaking glass, and protected or ordered targets. Tower varies Hanoi depth and board physics, then repeats boards on a dexterous hand. Home covers shelf tidying, mobile multi-step chores, tight insertions, and moving objects. A workload score cuts each type in half, into EP-Easy and EP-Hard of 50 tasks. The cut uses the task definition, so membership does not move when models improve. EP-20 takes 10 from each half. Every suite is one fifth of the subset score, so Launch cannot swamp Cube.
Each task has four prescribed conditions. They are perturbations, and they are separate evaluation instances. Launchers count only in loaded mode, first projectile already chambered, at spring stiffnesses of 22, 24, 26, and 28 N/m. Other families use seeds or lighting multipliers from 0.85 to 1.15, plus bounded pose, mass, and friction noise. Picking ammunition up off the ground is outside the score.
An independent scorer reads the executed trajectory. Goal, rules, release, and stability all have to hold inside the budget. Hanoi must rest for 1 second, lock assembly for 2 seconds and disassembly for 1, bookshelf rules for a continuous 0.5 seconds. Calling finish adds no settling time. Pass@1 is success on the first valid attempt. Pass@2 is success within the first two attempts on that same condition. Infrastructure faults do not spend an attempt. Ordinary failures and wall-clock timeouts do. The wall-clock cap is 10 minutes on launchers, 1 hour on Hanoi, and 19,800 seconds on a Rubik's cube, which may also use up to 109,800 actions.
The agents are GPT-6-Astra, run as Codex at high reasoning with L3 privileged observations, and Opus-5.5-extra at extra reasoning. Both write code. Neither gets auxiliary tools, reference solutions, or pretrained controllers. Each attempt may spend 30 million tokens, cached input included, and 100 images. All 400 conditions have a terminal outcome.
| Subset | Model | Pass@1 | Pass@2 |
| EP-100 | GPT-6-Astra | 13.80% | 19.94% |
| EP-100 | Opus-5.5-extra | 29.22% | 35.85% |
| EP-Easy | GPT-6-Astra | 18.42% | 27.41% |
| EP-Easy | Opus-5.5-extra | 33.99% | 42.88% |
| EP-Hard | GPT-6-Astra | 8.81% | 12.01% |
| EP-Hard | Opus-5.5-extra | 24.21% | 28.48% |
| EP-20 | GPT-6-Astra | 15.42% | 18.75% |
| EP-20 | Opus-5.5-extra | 30.58% | 39.13% |
Astra's first valid attempt solves 64 of 400 conditions. A second attempt adds 27, which is 8.04% of the 336 opening failures, and 309 conditions stay unsolved. A flat average across conditions is 16.00% and 22.75%. Equal suite weights pull the headline numbers down to 13.80% and 19.94%. Opus uses 683 valid attempts for 35.85% Pass@2; Astra uses 736. From Easy to Hard, Astra's Pass@2 falls from 27.41% to 12.01%, and Opus from 42.88% to 28.48%. Home breaks the pattern: Astra scores 19.44% Pass@2 on Hard and 12.50% on Easy.
| Suite | GPT-6-Astra | Opus-5.5-extra |
| Launch | 19.12% / 25.74% | 15.51% / 27.70% |
| Tower | 29.76% / 41.67% | 60.59% / 66.14% |
| Lock | 6.25% / 14.06% | 39.81% / 45.63% |
| Cube | 2.08% / 2.08% | 6.04% / 10.72% |
| Home | 11.76% / 16.18% | 24.17% / 29.06% |
Those cells are Pass@1 / Pass@2 on EP-100. Astra's best suite is Hanoi, at 41.67% Pass@2. Inside it, Physics reaches 68.75%, Plan 50.00%, and the dexterous-hand slice Dex 9.38%. Direct aiming reaches 46.43%. Rule-constrained launcher scenes reach 5.56%. All nine Lock successes sit in Easy, and the dexterous-hand lock records none. The 2.08% Cube score is one pocket-cube condition. No Rubik's cube is solved. Home successes occur only in the precise and dynamic types. Shelf tidying and the long mobile sequences score zero.
Opus reaches 66.14% Pass@2 on Hanoi, 45.63% on locks, and 10.72% on cubes. On launchers, Astra's Pass@1 is the higher of the two, 19.12% against 15.51%, while Opus is slightly higher on Pass@2. Of 84 Tower conditions, Opus solves 55 and Astra 35. Of 48 Cube conditions, the split is 5 to 1.
Time@S, mean wall-clock on the successful attempt alone, is 6.2 minutes for Opus on Launch, 26.6 on Home, 32.5 on Cube, 37.0 on Lock, and 38.5 on Tower. Astra's 91 successes average about 6.25 minutes. Provider token logs are incomplete, so Tok@S is missing.
Failures cluster. Among Astra's 645 scored failures, contact checks mark 97 of 115 lock attempts, mechanism constraints mark 64 of 94 cube attempts, and limit checks mark 129 of 211 launcher attempts, including 86 elevation-limit events and 50 shots fired before the aiming cycle finished. Clean sweeps are rare. Astra passes all four conditions on 4 tasks at the first attempt and on 9 tasks within two. Forty tasks have at least one condition succeed within two attempts.
The suite split is the useful part. Hanoi is partly solvable. Cubes and long household sequences are close to out of reach. That profile carries more information than a single 35.85%. Equal suite weights, fixed perturbations, and a separate bin for infrastructure faults give the next model a table it can line up against. The published run fills one cell: Astra with L3 privileged observations, no auxiliary tools for either agent, and no ablation. Two coding agents, allowed 30 million tokens and left to write controllers inside simulation, still finish under 40% suite-balanced Pass@2.
Of Astra's 645 scored failures, 574 (88.99%) ran into a rule or a numeric threshold that was not fully disclosed, including launcher joint-speed margins, forbidden-contact thresholds on the lock, and fixed-joint tolerances on the cube. Those trials stay failures. The headline rate is completion under the implemented contract. It does not, on its own, measure reasoning or control. Termination labels likewise leave open whether the agent, the controller, or the simulator produced the contact.
The comparison set is thin. There is no vision-language-action policy, no classical planner, and no L0-versus-L3 ablation. Astra still fails widely while reading privileged state, so missing vision is not the whole bottleneck, and the contribution of that state is unmeasured. Only two coding agents were run. There is no real robot. Half of EP-20's hard tasks had already succeeded once in pilot runs on the same or a closely related scene, so that subset sits optimistic. Gaps across suites are not a controlled comparison of reasoning type or embodiment. Sixty-nine infrastructure incidents were removed from the budget, and two native successes outside the scoring window were left unscored. A Rubik episode may last almost 5.5 hours and more than a hundred thousand actions, which mixes an unsuccessful search with an early reject from an undisclosed tolerance.