RoboICL: Embodied In-Context Learning with GPT-6 Astra
Fangcheng Liu, Yeqing Shen, Anda Cheng, Weishi Mi, Chao Tang, Chenyuan Liu, Yushun Xiang, Tingguang Li, Yong-Lu Li, Yehui Tang
cs.RO, cs.LG
2026-09-28
RoboICL feeds frozen GPT-6 Astra structured demos plus anchored interaction memory, zero robot training: 50.64 overall on RoboDojo vs 33.68 for the strongest baseline.
General-purpose VLMs can already emit robot actions zero-shot: GPT-6 Astra handles open-ended and language- or image-conditioned manipulation well, but falls apart on high-precision insertion and long-horizon tasks. The standard fix is a trained VLA, which costs large robot-specific data collection and post-training. A Samsung team (Samsung Research and Samsung Robotics eXperience, with Shanghai Jiao Tong University) flips the question: with weights frozen, can careful context organization let GPT-6 Astra make full use of demonstrations and its own execution experience? RoboICL is their answer.
Inference-time context is split into two sources that share one four-part grammar: observation, action, receipt, result observation.
Retention uses bounded anchored memory: B slots, the first B-1 at evenly spaced temporal anchors (anchor 0 always keeps the initial scene, which matters when the goal depends on the starting state), and the last slot holding the latest interaction for immediate correction. Discarded intervals get an explicit <TRAJECTORYGAP> marker so frames hundreds of steps apart are not read as adjacent. The reason is a hard 50-image cap per API request: uniform anchors keep early visual evidence from being evicted, the latest interaction preserves feedback, and memory stays bounded no matter how long the episode runs. An ablation backs the design: anchored memory beats a first-plus-recent-11 selector by 6.75 points (80.25 vs 73.50).
The action interface is a single act tool: the model outputs an H×14 matrix (7 values per arm, world-frame translation and rotation increments plus an absolute gripper target), deterministic code validates, solves inverse kinematics, and executes at 25 Hz, with H picked from the task step limit in five tiers from 5 to 25. Three camera views are stitched into one 1920×480 triptych, so one observation costs one image slot. Instruction, demos, and completed anchors form a stable prefix with cache-boundary hints; the measured KV-cache hit rate is 92.4% across 1,810 calls.
On 30 RoboDojo tasks, zero-shot for Open (no demos exist there) and one demo elsewhere:
| Setting | Metric | Score |
| RoboICL | 30-task Overall | 50.64 |
| Liber-0 Preview (strongest baseline) | 30-task Overall | 33.68 |
| RoboICL vs official zero-shot GPT-6 Astra | per-category progress | +20 to +27 in all four |
| RoboICL (ten-task panel) | Overall | 60.60 |
| GPT-as-Policy Direct (same panel) | Overall | 38.50 |
| π0.5 + GPT-6 Astra hybrid (same panel) | Overall | 62.60 |
By category: first on Memory (70.00) and Open (57.8), level with Liber-0 Preview on Precision (38.53 vs 38.28), third on Long-Horizon (44.11, behind 46.0 and 47.7). The most lopsided single task is Deposit Coin, a millimeter-scale insertion: every comparison method tops out at 18.67, while RoboICL averages 78.00 over 50 layouts with 37 full-score episodes, driven by wrist-camera corrections at millimeter scale. On a Franka Research 3 across three real tasks (five trials each), mean progress goes from 14.45 zero-shot to 63.33 one-shot and 78.89 three-shot. An optional Jev gate, which uses proprioception alone to decide whether to reuse an already-predicted action suffix, cuts GPT-6 Astra calls by 33% (181 to 121) and 48% (116 to 60) on two tasks, with tokens falling from 9.53M and 4.95M to 5.08M and 1.39M, though success moves in opposite directions (3/5 to 2/5, and 4/5 to 5/5).
This is the strongest evidence to date that context engineering alone can put a general-purpose VLM on top of a robot-control leaderboard: one shared grammar, anchored memory, cache-friendly prefixes, zero parameter updates. For teams without the data to train a VLA, it is the lowest-barrier route available, the code is open, and the interface details (receipts, gap markers, image budgeting) are directly reusable. The cost is equally plain: 2.6M to 5.6M tokens and 37 to 59 wall-clock minutes per episode. Fine for benchmarks and offline tasks, a long way from real-time control.
The authors' own list: one model (GPT-6 Astra), one simulation benchmark, three real tasks; the Jev gate was configured on the very layouts it was evaluated on, a development-set result; the allocation and memory ablations used tasks that also informed the design, so held-out validation is still needed.
The protocol gaps matter too. RoboICL runs seed 0 only and five layouts per task on the ten-task panel, while the published VLA/WAM baselines report three seeds and 50 episodes per task, so part of the margin is protocol. Some discontinued tasks are waived and scored zero; RoboICL's column carries five zero-score tasks, and the net effect on Overall is never broken out. Demos can hurt: Classify Objects drops from 100 zero-shot to 71 with one demo. Out-of-distribution generalization is the weakest point, as a larger unseen towel takes Folding Towel from 100 to 40, with the frozen model following the demonstrated motion instead of rescaling to the object. Long-Horizon is the one category it does not lead.