An Unexpected Robot Policy: Early Evaluations of GPT-6 Astra on RoboDojo and Beyond
Wenbo Zhang, Kaixuan Wang, Yutao Ouyang, Xiaoyu Huang, Liyang Li, Kailun Su, Weiyang Jin, Wenhao Chai, Haotian Liang, Zhiyang Dou, Yue Chen, Tianxing Chen
cs.CV
2026-09-21
Frozen GPT-6 Astra as a robot policy scores 22.48% on 2,100 RoboDojo trials, above all 40 public policies; GPT-5.5 scores 0.88%, and Astra's Precision SR is 4%.
Robot stacks usually split the job. A vision-language model (System 2) decides what should happen. A pretrained VLA or world action model (System 1) fires high-frequency motor commands. SayCan, Inner Monologue, Code as Policies, VoxPoser, Hi Robot, and Helix all sit in that pattern.
The question here is narrower: can a frozen frontier LLM issue the executable motion targets itself, with no learned policy in the loop and no robot or task fine-tuning. Call that LLM as policy. Prior evidence was thin. GPT-4 could emit end-effector trajectories, but it leaned on external detection and segmentation, on a 30-task suite. Anthropic reported rising subgoal progress on LIBERO-40, with full-task success stuck between 0 and 5.5%. GPT-6 Astra launch reports looked strong on selected real robots, on small custom suites and mismatched interfaces. None of that ranks an LLM against a full public policy board under one protocol.
RoboDojo simulation is the primary test: 42 tasks, five capability axes, 50 episodes per task, 2,100 trials for a full run, against a 40-policy public board frozen on 10 September 2026. Hardware is diagnostic only.
Each turn the model must call moveeef or giveup. moveeef names any subset of world-frame grasp-point position, orientation, and gripper for either arm. Unnamed dimensions stay as observed. Naming both arms produces a simultaneous motion. There are no pick, place, or pregrasp primitives, because those would finish part of the task outside the model.
A non-learned converter rejects invalid targets, maps grasp-point poses to flange poses, plans a joint path, and resamples it at 25 Hz. A rejected target returns an error and leaves the robot still, so the model retries on the same observation. An accepted target executes, then the next observation reports arrival error. Success is judged only by the environment.
Observations are three RGB views (head, left wrist, right wrist), 14-D grasp-point state, 12-D read-only joints, and the remaining step budget. Only the last two image turns stay in context. The Goal message adds a wiki-derived task recipe: what the task is and how partial credit is scored, not an action script. System notes include using an idle wrist camera. Default budget is 100 model calls per episode at medium reasoning effort. Astra and GPT-5.5 run the official 50×42 protocol. DeepSeek-Flash is 10 episodes per task, marked †.
Astra ranks first at 28.97 Score / 22.48% average success rate (micro 472/2100 successes, 28.72 Score), above all 40 public policies. The best public entry, DM0.5, sits at 24.90 / 19.34, 3.14 points of SR behind. With the same action interface, GPT-5.5 scores 1.13 / 0.88% and DeepSeek-Flash 2.99 / 1.92%. That is about a 25× spread among three LLMs. The wrapper is not doing the work.
| Model | Average Score / SR | Open SR | Precision SR |
| GPT-6 Astra | 28.97 / 22.48% | 31.00% | 4.00% |
| DM0.5 | 24.90 / 19.34% | 2.08% | 16.75% |
| GalaxeaVLA (G0.5) | 20.23 / 14.88% | 1.58% | 20.42% |
| GPT-5.5 | 1.13 / 0.88% | 2.75% | 0.00% |
The lead is concentrated. On Open, Astra hits 31.00% SR against DM0.5's 2.08%. On Generalization, 30.50% against 10.95%. On Memory, Precision, and Long-Horizon, Astra trails DM0.5: Precision 4.00% vs 16.75%, Long-Horizon 8.25% vs 19.50%, Memory 38.67% vs 47.44%.
Task-level splits are sharper. Astra reaches 60% on pushT and arrangelargestnumber, where the best public policies sit at 0.7% and 4.7%. alignblocks and solveequation are 50% and 40% for Astra, 0% for the public board. The other way: GalaxeaVLA (G0.5) scores 90% on makekong and 78.7% on buildtower; Astra scores 0% and 2%. DM0.5 reaches 59.3% on inserttubes; Astra is at 0%. Semantic matching and language-conditioned sorting work. Insertion, stacking under contact, pouring, and coordinated two-hand work do not.
One demonstration does not help. On 340 matched task–layout pairs across 34 tasks, zero-shot is 22.9% (78/340). Image plus end-effector demos fall to 17.9%. Text demos fall to 12.9%. Almost none of the recovery is on tasks that had zero zero-shot successes; the drop comes from tasks that already worked. Under perturbations on eight generalpickup layouts that Astra already solves: vertical flip 8/8, left–right mirror 6/8, no head camera 6/8, right-wrist only 3/8, 10 cm pose jitter 3/8, negated Cartesian axes 4/8. Selected traces show the model switching hands or reversing a direction after seeing the outcome. Eight binary trials per cell cannot separate robustness, retrying, and genuine adaptation.
The official RoboDojo-Real protocol was not finished. Repeated unsafe commands damaged equipment; no person was injured. The retained 33 diagnostic trials across 12 tasks score 6.97 / 3.03% SR, with one full success on stackbowls. That is a selected sample, not a hardware result. Outside the tabletop suite, a mobile humanoid walking-and-grasp clip has no score. A separate piano setup has the same frozen model write a 45-D controller at 20 Hz: one-hand Twinkle F1 0.907, two-hand 0.902 (RL replay reference 0.886 in this setup), a Chopin nocturne excerpt 0.599. Those runs are code synthesis with practice episodes, not LLM-as-policy.
For people training VLAs, the concrete claim is that a frozen general LLM can outrank the current public policy board on RoboDojo's semantic and open-ended axes, with no robot fine-tuning. That is a real crack in the "LLM only plans, VLA only acts" default.
The crack stops at contact. A 4% Precision SR, hardware damage, and a one-shot demo that lowers success are not a replacement for a 25 Hz motor policy. A more honest split: send language- and image-conditioned tasks to this class of model; keep insertion, pouring, and dynamic grasping on a VLA or WAM. The interface is cheap: Cartesian targets plus geometric IK, no extra action head.
The other hard fact: GPT-5.5 and DeepSeek-Flash barely move the robot on the same tools. The capability is model-specific. "LLM as policy" is not a free lunch that any frontier API inherits.
The paper is explicit. Deep analysis covers Astra only. Main runs and ICL probes use one seed. Three models cannot set a class-wide threshold, and DeepSeek-Flash has 10 episodes per task. LLMs also receive a wiki recipe and process-score ladder that public policies never saw, so the comparison shares the evaluator, not the inputs. moveeef is quasi-static, RGB has no metric depth, and Memory is entangled with a two-frame image horizon. Interface ceiling and model failure are not separated.
The 33 real-robot trials are a selected sample after a safety stop. Contamination of a closed model on benchmark-related text is unquantified. The perturbation table is eight binary trials per cell on layouts chosen for unperturbed success, so the no-perturbation row is at the ceiling by construction. The one-shot drop could be a precision bottleneck or layout-geometry copy; the experiment does not tell them apart. The ranking and the capability split come from one simulator. The official real protocol was not completed, so neither result is shown to transfer.