From VLM Planners to Robotic Agents: Self-Evolving Harnesses for History-Dependent Manipulation
Tingwen Du, Haoyu Guo, Hanxue Zhang, Xuanxu Wang, Junting Dong, Chao Gao, Mingfei Han, Tao Lu, Mulin Yu, Chunhua Shen, Xiaojun Chang, Zhihui Li
cs.RO
2026-10-21
SERA freezes the VLA and edits a harness around gpt-5.6-sol. On 16 RoboMME tasks, selected harnesses reach 51.38% success, up 7 points from 44.38%.
Long-horizon manipulation often shows the robot the same picture for two different next moves. A button looks similar after one press and after two. Whether to press again depends on how many presses already succeeded. The order of locations in a demonstration, or a cue that has left the frame, is also absent from the current image. The decision is non-Markovian: the instantaneous observation does not pick the action. Partial views and execution error then accumulate.
Adding memory to a VLA, a vision-language-action model that maps images and language directly to motor commands, is one existing route. Splitting a VLM planner from a VLA executor is another. Neither gives the planner its own books. Sending a command does not establish that the step worked. Progress has to be rewritten from later images and feedback. Across attempts, a trace also has to become a change the next plan can reuse. Fresh demonstrations are expensive, so a new supervised set for the high-level VLM is a costly way to get that change.
SERA places the high-level VLM inside an editable harness, the outer task manager that assembles inputs, keeps context, checks commands, and carries feedback into the next round. The low-level VLA stays frozen and only turns a command into an action chunk. The planner is gpt-5.6-sol, called through Codex with reasoning effort set to high.
At episode start the harness writes a fixed task brief from the instruction and any initial context, which may be a demonstration. During execution it rewrites a progress summary, capped at 1,600 characters, using the front camera, the wrist camera, and feedback. The summary records phase, completed steps, the grasped object, gripper state, and unresolved uncertainty. The text stays after those frames leave the recent window.
Each planning call receives the brief, recent images, the running context, the previous command, and action candidates. Candidates are drawn from the executor skill catalog. Reciprocal-rank fusion over the previous skill, recorded successors, and frequency supplies the top five, with no semantic similarity. In the same call the VLM judges the last command and returns an updated context plus the next command. Once the template and grounding points pass a check, the frozen VLA executes a bounded prefix of its action chunk. Stagnation counters can force a fresh look or a new command. The visual check has no separate code-level veto, and the progress fields are model estimates.
Self-evolution edits the harness rather than the episode context, and it does not build a supervised fine-tuning set for the VLM. An analysis agent localizes the first divergence and separates motor failure from progress-reasoning failure. Revisions touch only planning guidance, assessment criteria, and control rules. A verifier compares each candidate with cached baseline outcomes on the same development episodes, then continues, switches repair target, or stops. The selected harness is frozen before held-out testing. Diagnosis may consult expert demonstrations or benchmark traces. A deployed trigger may use only task identity and information visible at runtime, and the checked runtime rejects unregistered code plugins.
Evaluation uses RoboMME: 16 history-dependent tasks in Counting, Permanence, Reference, and Imitation, 50 episodes each, 800 total. Success is an unweighted average over tasks. Baselines in the comparison are public leaderboard scores. SERA reports the autonomously selected harness.
| Method | Overall success |
| π0.5, current view only | 17.93% |
| π0.5 plus action history | 19.40% |
| SAM2Act | 21.37% |
| FrameSamp+Modul | 44.51% |
| MemER | 42.38% |
| SERA | 51.38% |
On Reference and Permanence, SERA scores 69.50% and 57.00%, the best numbers in Table 1. MemER scores 38.00% and 53.17% there. FrameSamp+Modul scores 36.33% and 25.11%. Counting is a loss: SERA at 49.00% versus FrameSamp+Modul at 65.22%. Imitation is a loss too: 30.00% versus 51.39%. The overall lead coexists with two suite losses. Other hierarchical entries sit lower still: SimpleSG+QwenVL 29.00%, GroundSG+QwenVL 32.70%, GroundSG+Gemini 11.56%.
Without self-evolution the same harness scores 44.38%. With it, 51.38%, up 7.00 percentage points. Figure 3a splits the gain. Reference goes from about 69% to 69.5%, Permanence from 56% to 57%, Counting from 34% to 49%, and Imitation from 18.5% to 30%. Keeping a demonstration target and a running execution state is mostly the strong planner plus context. Self-evolution repairs rule gaps in counting and imitation, such as a reversed direction map and action candidates that omit commands the executor already supports.
On a six-episode PatternLock development panel, success moves from 2 to 3 after the direction fix, then to 5 after all eight directional commands are exposed. The paper states that this is not an isolated comparison of the two repairs. The test result on that task is 30/50. A local MemER rerun scores 363/800 against SERA at 411/800, distinct from the leaderboard 42.38%. The large per-task gains are VideoPlaceButton (49/50 versus 14/50), VideoPlaceOrder (46/50 versus 18/50), and VideoUnmaskSwap (46/50 versus 18/50). ButtonUnmask goes the other way, 20/50 versus 41/50. ButtonUnmaskSwap, InsertPeg, and RouteStick are all 0/50.
The practical takeaway is limited and concrete. Do not fine-tune the planner. Keep bookkeeping, command checks, and post-failure rule changes in an outer harness, revise that harness from rollouts, and leave the VLA frozen. That touches less than collecting a new demonstration set for supervised fine-tuning.
The fixed-planner comparison is 44.38% to 51.38%. The gap between 51.38% and FrameSamp+Modul at 44.51%, or MemER at 42.38%, is not. SERA plans with gpt-5.6-sol. Leaderboard hierarchies plan with QwenVL, Gemini, or MemER's own high-level model. Reference was already near 69% before any harness revision.
Patches may activate by task identity, and a task with no selected revision keeps the base harness. After the search budget is exhausted, a fallback with one extra development success and no observed regression can still be retained. Selected patches remain in the reported average even if the test score falls. They are not replaced by the better of base and adapted once test outcomes are known. The 7 points summarize that selection procedure.
The paper leads its own caveats with planning latency. Every round runs VLM inference over images and context. On a long horizon the overhead adds up and can miss the timing of a dynamic task. The exchange between reasoning quality and responsiveness is not measured. No latency figure is reported.
Planner intent still fails at the executor. In an exploratory RoboMemArena task that needs two pours and then a rack placement, the planner requests the second pour and the executor racks the container. Context keeps the unfinished requirement. Later regrasps fail. Gripper interventions on development runs sometimes finish the task when the matched run without them fails, and sometimes fail too. Reliable recovery stays open. The current harness only covers the planner side.
Diagnosis may use expert demonstrations, so the absence of a fine-tuning set does not mean evolution never saw expert traces. Three test tasks score 0/50, and Reference carries the average. The frozen VLA checkpoint is never named, so equality with the leaderboard executors cannot be checked. The benchmark is simulation only. The progress summary is a model estimate, and an error in it is what the next planning call reads.