From Pretraining to Proficiency: Real-World Subtask RL for Long-Horizon Manipulation with Minimal Human Intervention
Sichang Su, Benjamin Yang, Zhiyun Deng, Boyuan Liang, Yip Fun Yeung, Zelin Wang, Lingfeng Sun
cs.RO, cs.LG
2026-09-18
PARTS trains residual RL only at bottlenecks of frozen π0.5, lifting YAM success 32% to 61% and Franka 50% to 95% in tens of minutes per task.
A pretrained VLA can often walk through most of a long-horizon job and still fail at a few bottlenecks: millimeter earbud insertion, a cable that must hit a port, or LEGO bricks smaller than the demonstration set. Collecting more full-task demos for SFT makes the operator repeat steps the policy already handles. Full-task RL is an awkward fit. The reward arrives only after 20–120 seconds. A good intermediate step still scores zero if a later one fails, and early failures starve later stages of practice.
The need is to spend robot time on the bottlenecks, keep humans off the loop during rollouts, and still grade the complete task.
PARTS (Policy Adaptation with RL on Targeted Subtasks) freezes the base policy and attaches residual actors only at human-marked bottlenecks.
Start from π0.5, SFT it on task demos to get π0, and watch where it fails on the robot. A person writes a contract per bottleneck: entry states, which action axes may move and their bounds, a success test, a motion budget, plus a few subtask demos. Coding agents (Claude Fable 5 and GPT-6 Astra) turn that into three programs:
SAM3 supplies object masks. Gemini-3.7-flash answers asynchronous questions about object state. Each residual is a small actor-critic over the base visual features, proprioception, and the nominal action chunk, outputting a bounded correction masked to the relevant axes (gripper opening, end-effector translation). Training is chunk-level TD3+BC: clone residuals from successful attempts, pull failed residuals toward zero. Reference dropout on the nominal chunk stops the residual from copying the base.
Online updates see few rewards when base success is low, and restaging to the subtask start (to save reset time) narrows the state distribution versus evaluation. Periodic success-reweighted retraining keeps every success and a fraction ρ of failures, trains fresh networks on that set, and redeploys them to collect more data. At eval, residuals take over at their entries and hand back at their exits.
Humans are pushed off the rollout. DSRL rolls autonomously but needs people for reward and reset. EXPO-FT and RLT involve a human handoff or intervention every episode. PARTS rollouts need no action corrections or policy switching. LEGO and cable rewards and resets can run unattended; earbuds still need a human check of the verifier, and a human pickup when a bud hits the floor.
Three long-horizon tasks, 20 full episodes per method, matched robot-rollout time. Bimanual YAM: open a charging case, hold it, insert two earbuds, close the lid; and sort ten small bricks into three color bins in 150 s. Single-arm Franka FR3: unplug a cable from one router and plug it into another. Bottleneck windows are typically 3–15 s.
| Task | Base (SFT) | PARTS | Rollout time |
| YAM full-task success | 32% | 61% | tens of minutes / task |
| Franka full-task success | 50% | 95% | 29 min on cable |
| LEGO progress (fraction of 10 bricks) | 54% | 82% | 17 min |
Earbud full-task success is about 4× the base, with each insertion up 20–35 points. Cable unplug already works, so practice goes to insertion, which nearly doubles. Baselines spend the same robot time on full-task RL: DSRL searches in the frozen VLA’s noise latent space, EXPO-FT also updates the backbone, RLT hands a single RL policy the rest of the episode after a human-chosen switch. PARTS beats these by more than 25 full-task points. EXPO-FT falls below the base and sometimes breaks a working unplug. Dropping success-reweighted retraining hurts both subtask and full-task success on earbuds.
This is a repair kit for “the base policy mostly works, a few precise steps do not,” not a from-scratch skill learner. Freezing the VLA, using local rewards, and limiting residuals to a few axes shortens credit assignment from a minute to a few seconds, and it keeps sparse full-task rewards from overwriting stable prefixes.
The setup cost is real: people still mark bottlenecks and write contracts, then review agent-written selectors and verifiers. Earbuds, where a seated bud and one resting on the rim look alike, still need a human eye on the verifier. Unattended overnight RL is not this system.
Twenty episodes per task is thin for a binary long-horizon metric; 32% → 61% is a magnitude, not a ranking you should over-read. Only π0.5 is used as the backbone. LEGO’s failure is largely “the gripper does not close enough,” so the residual is close to a one-dimensional fix; messier contact-rich assembly is untested.
Humans did not leave: bottleneck discovery, contracts, program review, some resets, and earbud labels remain. The paper’s own future work asks agents to find bottlenecks, write denser rewards, and add recovery phases. Retraining still has an operator picking checkpoints. The coding agents are specific model snapshots; whether a later model writes stable selectors and verifiers is unmeasured.