RoboCoach: World Models as Active Coaches for Compositional Robot Skills
Jiajun Liu, Yifan Chen, Yichao Liu, Jiayi Zhang, Ruoqu Chen, Shaoxuan Xie, Guocai Yao, Mengdi Xu, Sen Cui, Changshui Zhang
cs.RO, cs.AI
2026-09-30
RoboCoach rolls out skill experts inside a world model, pins down the first failing subtask, and requests targeted demos; 150 extra demos lift Franka success from 13.3% to 75.0%.
Long-horizon manipulation tasks such as preparing tea or setting a table are chains of reusable atomic skills, and each skill changes the state the next one faces, so one local execution error can cascade into total failure. The default fix is more end-to-end demonstrations, but on a physical robot every trajectory costs collection labor, hardware time, and environment resets, and the space of skill compositions is a long tail you cannot cover. The question becomes concrete: the next demonstration is expensive, so which skill should it teach, and which component of the policy should receive it?
The ingredients already existed: generalist VLA models for execution, active imitation learning for data selection, world models for imagined rollouts, modular policies for piecewise updates. What was missing was the wiring between them: a predicted failure has to turn into an actual request for demonstrations, and those demonstrations have to land on the policy component tied to that failure.
RoboCoach organizes coaching into a Route–Imagine–Diagnose–Improve (RIDI) loop:
The design choice that matters is pairing shared with selective: prediction is shared (one CoachWorld serves every task and embodiment) while updates are selective (demonstrations land only on the expert that failed). What to teach and who learns are bound to the same unit, which is where the gains come from.
First, is the world model qualified as a judge? Across 22 frozen task–policy pairs, imagined and deployed success rates correlate at Spearman 0.840. RoboMeter reaches a switch MAE of 0.414 s and an outcome F1 of 88.21% on reference video, dropping to 0.815 s and 83.84% on CoachWorld-generated video, still serviceable. On DROID-180, CoachWorld posts the best LPIPS (0.0996), FVD (51.71), and instruction-following score (0.880) among the compared models.
Then the coaching gains (complete-task success, real robots):
| Setting | Method | Start | After 150 demos |
| Franka | RoboCoach | 13.3% | 75.0% |
| Franka | Uniform + shared adapter | 13.3% | 30.0% |
| AgileX | RoboCoach | 40.0% | 83.8% |
| AgileX | Uniform + shared adapter | 40.0% | 47.5% |
The ablations locate the source of the gain (simulation, matched budgets). Feeding the targeted demos into a shared adapter instead of the matching experts moves final success by just +0.8 pp on LIBERO and −4.8 pp on RoboTwin; routing them to the matching experts adds +3.4 and +13.2 pp, and world-model targeting beats random pair selection by +2.8 and +12.8 pp. Demos landing on the right expert does the work; selection alone contributes little. Final success reaches 71.2% on LIBERO and 68.0% on RoboTwin, the highest of the four conditions.
On four held-out compositions (longer continuations, cross-task combinations, skill reordering, never seen in training or coaching), RoboCoach averages 35.0% (65%, 15%, 25%, 35% per route); the uniform-plus-shared-adapter baseline scores 0% on all four.
For teams collecting robot data, this hits the actual bottleneck: not how to collect but what to collect. Rolling out in the world model to find the failing skill, then collecting that skill specifically, puts the demonstration budget where it matters, and the coached skill stays reusable across compositions. The exchange rate, 150 demos for 75% success on Franka, is a directly comparable anchor for real-robot data budgets.
The broader point is the role shift of world models. Most work uses them to generate more experience by imagining training data; RoboCoach uses one to allocate real experience, as a diagnostic and scheduling layer. Imagine first, then decide what to learn next transfers to any setting where data is expensive.
The authors state three: prediction degrades when the manipulated object is occluded or contact happens outside the camera view; multi-seed voting suppresses stochastic variation but cannot rule out systematic world-model bias; the expert library is predefined by skill semantics, with automatic library learning and removing the reliance on human demonstrators left open.
Two things stand out on a close read. The 0.840 correlation is task-level; the paper never directly reports how often the subtask flagged as the bottleneck in imagination matches the real bottleneck behind the 13.3% to 75.0% jump, so a systematically optimistic CoachWorld would skew data allocation. And 35.0% on held-out compositions is middling in absolute terms, with the cross-task composition route at 15% showing that recombination remains hard. Demonstrations are still collected by human operators from a list; the automation here stops at generating the request.