Recursive Harness Distillation across Agents for Robot Manipulation
Seungyeon Kim, Junhoo Lee, Minkyu Kim, Baekseung Kim, Nojun Kwak
cs.RO, cs.AI, cs.CV, cs.LG
2026-09-27
Playbook distilled from a strong agent and refined on light-agent rollouts lifts real-robot success from 37.3% to 64.0%, and Bridge success from 41.7% GR00T-only to 66.7%.
VLA policies map images and language to robot actions, and they already cover many skills. Coverage does not tell the system what to do when the current scene and the intended outcome diverge. Prior work lets language models write robot programs, or treats learned policies as tools inside a planner that monitors and recovers. Naming the right subgoal still leaves an open question: how will this deployed VLA respond to this image, and which knob actually changes its behavior.
A strong model can find those knobs by interacting with the policy. The paper uses GPT-6 Astra. Running it for every episode is expensive. A cheaper model can sit at the same interface and still fail to use it. RHD freezes the VLA and both agents, and stores the knowledge as an external playbook: when to intervene, how, and what to check afterward.
Recursive Harness Distillation builds a harness: an intervention interface plus a playbook written in natural language. No weights move. Experience lives in the playbook.
The interface exposes three sites on the frozen VLA:
The agent can touch any subset. It can inspect a candidate without stepping the robot.
The loop is teacher writes, student runs, teacher revises from the student's failures. The strong agent first operates the VLA and writes an initial playbook P0, here from Astra's first eight development rollouts. The light agent then executes with that playbook. The strong agent compares teacher and recipient traces on the same development cases, then rewrites applicability conditions, intervention choices, and outcome checks. A candidate is accepted only if its success on the development batch reaches the teacher's rate η. If none reach η, the playbook is not updated.
The objective is the light agent's task success under the playbook. A revision should improve decisions at histories the recipient actually visits. Copying the teacher on the teacher's own histories does not measure that quantity. The same mismatch shows up in sequential imitation methods such as DAgger.
Simulation uses four WidowX tasks in SimplerEnv Bridge: spoon-on-towel, carrot-on-plate, cube stacking, and eggplant-in-basket. The policy is frozen GR00T-N1.7-SimplerEnv-Bridge. Twelve initial configurations for development, twelve for evaluation, 48 instances per split. GPT-5.6 Luna is the light agent. Evaluation configurations never enter playbook writing.
| Method | Success |
| GR00T only | 41.7% |
| Luna + GR00T, no playbook | 43.8% |
| Luna + initial playbook | 31.3% |
| Luna + refined playbook | 66.7% |
| Astra + refined playbook | 79.2% |
Adding Luna without a playbook moves success from 41.7% to 43.8%. The first playbook drops Luna to 31.3%. After refinement Luna reaches 66.7%, 22.9 points above its unguided run, and beats Astra without a playbook. Astra with the same playbook reaches 79.2%.
Ablations keep the refined playbook and disable one site: no instruction edit 47.9%, no attention edit 64.6%, no action edit 62.5%. Instruction rewriting is the largest slice.
A Luna teacher writing and revising for a Luna recipient gets 22.9%, below the 43.8% no-playbook baseline. Self-distillation from the light model produces a harmful playbook.
Real robot: 7-DoF Franka Panda, three tasks (cube-to-tray, cube stacking, button pressing). π0.5 is fine-tuned on 50 teleop demos per task and then frozen. Objects are randomized in a 50cm × 50cm workspace. 25 trials per task, 75 total. Astra adapts the GR00T playbook to π0.5 before Luna uses it.
| Method | Tray | Stack | Button | Overall |
| π0.5 | 15/25 | 3/25 | 10/25 | 37.3% |
| Luna + π0.5, no playbook | 0/25 | 0/25 | 0/25 | 0% |
| Luna + refined playbook | 18/25 | 13/25 | 17/25 | 64.0% |
Unguided Luna scores zero on the hardware. The paper says it could not correct recurring spatial offsets between the intended contact and the executed motion.
On the same Bridge test set, estimated API cost is $30.03 for Luna ($0.63 per episode) versus $164.61 for Astra ($3.43 per episode), about 5.5×. Surviving playbook rules include re-identifying the object after occlusion, and inspecting outputs because an OPEN instruction can still return CLOSE.
This takes the LLM-agent habit of writing experience as text and handing it to the next agent, and applies it to robot manipulation with a frozen VLA. For a policy you cannot keep fine-tuning, a strong model can discover interventions during development and a cheap model can carry the playbook at deploy time.
The same playbook also lifts Astra to 79.2%. What got stored is how this VLA fails and how to recover.
The first draft of the playbook hurts, and Luna teaching Luna is worse. The method is tied to a strong teacher and to recursive revision.
The paper has no limitations section.
The split varies initial poses inside the same task family, not new tasks. Real-world evaluation is three short tasks and 25 trials each; cube stacking's policy-only baseline is 3/25, so the percentage is noisy. The hardware playbook is adapted by Astra onto π0.5, so zero-shot transfer across embodiments and policies is untested.
There is no comparison with fine-tuning the VLA or with RL on the same interface, so the 26.7-point real-robot gain is not priced against a weight update. The $0.63 per episode excludes playbook development. The acceptance rule may never fire. Whether GPT-6 Astra and GPT-5.6 Luna can be rerun through the same interface is unclear. Attention interventions need access to internal VLA computation, which many open VLAs do not expose.