World In-Context Learning for Agentic Robot Manipulation
Heng Fang, Zehui Guo, Yuehan Zhang, Mingyang Du, Dingkang Liang, Xiang Bai
cs.CV, cs.RO
2026-11-10
With one demo per task and no gradient updates, RoboScript rehearses in simulation and reaches 83.4% mean success on 50 RoboTwin tasks, 21.7 points above 4D-WAM.
A manipulation skill stored in network weights stays tied to the scenes it was trained on. Vision-language-action models (VLAs) map images and language straight to motor commands. World action models (WAMs) predict a future scene or a latent plan, then read an action out of it. Both train with gradients on dozens of demonstrations, so a new task or a new visual distribution means new data and another training run. FastWAM, trained on clean RoboTwin 2.0 demonstrations, reaches 77.8% on clean scenes and 1.9% once textures, lighting, table height, and distractors are randomized together.
In-context learning leaves the weights alone and tries a new task from a demonstration in the prompt. That clip was recorded on a different table, so the prompt never says what the next action will do here. Errors stack along the steps. Coding agents avoid the same trap because a terminal returns the result of running the code. Manipulation has not had that loop.
What sticks in the context is semantic: which object plays which role, the order of steps, and what failed last time. Texture and lighting can change and those notes still apply. A simulator that is cheap, resettable, and faithful to the task is enough to rehearse in.
World In-Context Learning (WICL) writes world feedback into the context of a frozen foundation model. The model drafts a context from one demonstration, acts in rehearsal scenes, and rewrites the context from trajectories and action receipts. Rehearsal instances and test instances do not overlap, so a frozen context is scored on scenes it never practiced. The context is readable text, and evaluation logs can become guidance for the next round without a gradient step. With no world in the loop, WICL falls back to ordinary in-context learning from the demonstration alone.
RoboScript runs the loop in the RoboTwin 2.0 simulator. The learned context is a Task Memory, a Markdown file of object roles, stage order, relative grasp and place specs, and lessons. A spec reads like "approach the handle from the left and grasp its midpoint." It does not store a coordinate from one scene. A linter rejects coordinates, ground-truth terms, and instance ids.
Perception matches a real robot: head and wrist RGB-D plus proprioception. Object identity is a vote among Set-of-Mark marks, a grounding model, and a vision-language model, then tracking follows the grasp so occlusion does not force a fresh detection every step. Shared action entry points carry no environment-specific frames. A binding fills in joint limits, table height, and the gripper. A coding agent writes skills only for demonstration segments those entries cannot explain. A fit is thrown out if its parameters move more than 1 cm when the object is translated, which stops a skill from memorizing an absolute pose. The skill set is frozen before task learning. Every action returns a receipt from proprioception and the motion planner, with the pose actually reached and whether an object is held, so the model can separate a command that ran from an effect that happened.
Imitation turns the single demonstration into a first memory and practices on that same instance until attempts without hints pass. Rehearsal moves to new training scenes and counts only the first attempt on each. On failure, a reviewer sees a ground-truth diagnosis, edits a cached copy, and merges the edit only if the retry succeeds. The edit budget shrinks with practice. During rehearsal and deployment the acting agent sees only deployable sensors. Imitation roles may read privileged state, and that supervision reaches the actor only through a memory that passed the linter.
At deployment the memory is frozen and the world no longer returns a score. The agent maps roles onto the current objects, picks an arm that can reach, and on failure moves to a safe pose and continues without resetting, inside the official step limit. A self-declared finish has to match the demonstration's final state. The simulator's success check is the judge. Reflection reads disjoint blocks of those episodes, using only deployable observations and success or failure, and writes guidance for the next unseen block.
The benchmark is all 50 dual-arm tasks in RoboTwin 2.0. Trained policies see 50 clean demonstrations per task, trained jointly, and are tested for 100 episodes each. RoboScript gets one clean demonstration per task, rehearses only on clean scenes, and is tested for 10 episodes per setting. Retention is randomized success divided by clean success.
| Method | Demos | Clean | Rand. | Mean | Retention |
| RoboScript (GPT-6.1-Sol) | 1 | 83.6 | 83.2 | 83.4 | 99.5% |
| RoboScript (GPT-5.6-Luna) | 1 | 79.8 | 76.0 | 77.9 | 95.2% |
| RoboScript (DeepSeek-V4.1-Flash) | 1 | 77.8 | 78.4 | 78.1 | 100.8% |
| 4D-WAM | 50 | 81.5 | 41.8 | 61.7 | 51.3% |
| π0.5 | 50 | 70.7 | 46.0 | 58.4 | 65.1% |
| FastWAM | 50 | 77.8 | 1.9 | 39.9 | 2.4% |
| HarnessVLA (Claude Code) | N+1 | n/a | 58.4 | n/a | n/a |
GPT-6.1-Sol averages 83.4% (83.6 clean, 83.2 randomized), 21.7 points above the strongest trained policy, 4D-WAM at 61.7% (81.5 clean, 41.8 randomized). The 95% Wilson interval on Sol's mean is 81.0 to 85.6. On clean scenes the gap is 2.1 points. The best retention among trained policies is π0.5 at 65.1%. The three foundation models retain 95.2% to 100.8% of their clean success without ever rehearsing a randomized scene. Sol's randomized score is 24.8 points above HarnessVLA with Claude Code, which reports only the randomized setting. DeepSeek-V4.1-Flash and GPT-5.6-Luna average 78.1% and 77.9%, still above every trained policy in the table. Sol leads them by 5.3 and 5.5 points. Task-paired 95% intervals start at 0.7 and 0.8, while a change of API session can move success by several points on its own. Every π0.5 task category loses 18.7 to 31.8 points under randomization. Sol's category-level change stays between -7.0 and +6.4 points.
Of Sol's 166 failed episodes, knowledge failures are 37 on clean scenes and 38 on randomized ones. Perception failures fall from 22 to 17. Execution failures rise from 23 to 29. The agent's own completion check wrongly accepts an official failure about 28% and 29% of the time.
Ablations use 10 easier tasks and paired seeds, so the levels are not comparable to the 83.4% headline. Writing a memory from the demonstration without entering the simulator scores 69.0%, against 78.5% when the run interacts with the world (p = 0.003). Relearning with no demonstration drops 86.0% to 56.5%: three tasks still finish, two score 0. Removing privileged supervision leaves 86.5% against 86.0%. Ending an episode at the first failed action drops deployment from 78.5% to 69.0%. On 11 harder tasks, imitation without rehearsal scores 46.1%, the full loop 54.5%, and the randomized gap is 13.3 points (p = 0.034).
Three rounds of reflection on 17 weak tasks (at most 8 of 10 clean successes beforehand) net +18, +8, and +4 episodes. Across all 50 tasks the nets are +18, +7, and +1. Guidance entries grow from 74 to 112.
Learning all 50 tasks and evaluating 1,000 episodes costs 308 USD with Luna, 739 USD with Flash, and 1,911 USD with Sol at list API prices. Evaluation is most of the bill. A Sol episode averages 54.7 calls, 2.33 million tokens, and 149 simulation steps, with a median of 6.4 minutes, about 80% of it spent waiting on the model.
On a physical dual-arm PiPER, whose kinematics differ from the simulated ALOHA, an adapter remaps coordinates, arm parameters, and the gripper. Demonstrations are not re-recorded and rehearsal is not repeated. The action sequence is planned offline. Four tasks, 10 episodes each: Click Bell 10/10, Grab Roller 7/10, Move Playingcard Away 7/10, Stack Bowls Two 6/10. Failures are mostly gripper misalignment and real depth error.
On clean scenes, one demonstration plus rehearsal ties 4D-WAM trained on 50 demonstrations. The 21.7-point lead on the average is mostly retention once the scene is randomized. The skill lives in readable text, so swapping the foundation model does not require retraining.
The loop needs a world that matches task semantics, can be reset, and keeps rehearsal off the test instances. Without a simulator, that path is not available. A 6.4-minute episode can produce a memory offline. It is a poor fit for a servo loop.
API quality drifts, and a new session can move success by several points. Reflection gains shrink each round, and the remaining failures are hard to pin down from a sparse success bit. Every evaluated task uses rigid or articulated objects. Cloth, cables, and tight two-arm contact stay open. A single episode also waits on dozens of model calls.
Clean scenes account for only 2.1 of the 21.7-point gap in the mean. Trained policies are scored on 100 episodes per task. RoboScript uses 10 per setting. Resampling tasks widens Sol's interval to 77.5 to 89.0. The same memories on four further clean seed blocks land between 83.0% and 85.2%. Seed noise looks smaller than the spread across tasks.
"One demonstration" describes the task memory. The skill library has already seen demonstrations from all 50 tasks. The privilege-free ablation covers only 10 easier tasks, where it ties the full method. The real-robot study is four tasks, run open-loop, 10 episodes each. The specs transfer across embodiments. These episodes are too few to estimate a physical success rate.