AgenticRoboRL: Bringing Coding-Agent Post-Training to Agentic Robotics
Xinxin Liu, Bingxin Xu, Wenrui Bao, Ming Li, Wentao Bao, Chen Chen, Yuzhang Shang
cs.RO
2026-10-21
SFT plus multi-turn RL lifts a 9B robot planner from 18.1% to 71.0% on LIBERO-PRO, close to Codex (GPT-5.5) with memory at 72.1%, with all execution components frozen.
Agentic robotics systems pair an LLM planner with a robot harness: the planner decides what to do next, the harness turns those decisions into motion. Recent progress has been on the harness side, with better tool interfaces and external memory, while the planner seat goes to a strong closed model like Codex or Claude Code that is never trained for the job. Coding agents took a different path: post-training the model itself with SFT and RL. The question here is whether that recipe transfers. The obstacle is concrete: harnesses disagree on execution. Some have the planner invoke learned VLA skills, others have it write control code, and each returns feedback and judges success differently, so no single low-level action interface can serve a shared training pipeline.
Two stages, SFT then multi-turn RL, updating only the planner. The student is Qwen3.5-9B, the teacher Qwen3.8-27B, and the VLA, motion planner, and perception modules stay frozen.
SFT has a wrinkle: the teacher runs with an external memory module the student will not have. Raw teacher trajectories would teach memory dependence, so the framework flags responses that lean on memory (keyword hits, an 8-word verbatim repeat, numbers appearing only in memory results), rewrites them, and keeps only rewrites a judge approves. This step carries real weight: dropping memory-dependent responses yields 55.2%, rewrite-and-filter reaches 60.2% at a comparable optimizer step count, and the selected checkpoint hits 65.1%.
The RL design puts the shared interface at the model-call boundary, not the action boundary. A proxy records every model call's context, token IDs, and logprobs; a parameter-free harness adapter matches those records to episode outcomes via five fields (task, initial state, attempt index, reward, validity). One RL pipeline then serves both VLA skill invocation and control-code generation without touching the harness's tools, prompts, or control loop. The reward is terminal task success only, with no intermediate shaping and no length penalty; advantages use RLOO, and episodes that fail from infrastructure errors are excluded rather than scored zero. Each round collects 384 episodes (64 task/state groups, 6 attempts each), and the reported checkpoint took 2,304 episodes total.
The data split is clean: LIBERO gives 50 initial states per task, state 0 builds memory under the Harness VLA protocol, states 1-10 are evaluation, 11-44 supply teacher trajectories and RL rollouts, and 45-49 only validate checkpoints.
LIBERO-PRO, 800 episodes:
| Configuration | Success rate |
| 9B base, no memory | 18.1% |
| 9B base + external memory | 18.9% |
| 9B + SFT | 65.1% |
| 9B + SFT + RL, no memory | 71.0% |
| 27B base, no memory | 57.9% |
| 27B + memory (teacher) | 74.0% |
| Codex (GPT-5.5) + memory | 72.1% |
| Claude Code (Opus 4.7) + memory | 82.4% |
| Standalone VLA (πRLinf) | 50.0% |
SFT contributes 47 points of the gain and RL adds 5.9. On paired episodes, RL solves 121 that SFT fails and loses 74 (sign test, p = 0.001). Bolting external memory onto the untrained 9B moves it only from 18.1% to 18.9%: memory does not rescue a weak planner.
The same checkpoints go to LIBERO-Plus, 10,030 perturbed configurations, with zero additional training: 19.2% to 66.7% to 72.2%, improving across all seven perturbation categories. On robot initial-state perturbations it reaches 75.2% against 46.2% for the best compared method.
On the second harness, CaP-X, where the planner writes control code, the 9B goes from 2.7% to 18.7% after SFT and 28.5% after RL, past the 27B teacher's own 20.3%.
This runs the coding-agent post-training playbook end to end in robotics, at the framework level: an adapter only aligns recorded calls with outcomes, so a new harness plugs in without touching execution. Teams can swap a per-token closed model for a local 9B planner and keep their execution stack unchanged. The student beating its teacher on CaP-X (28.5% vs 20.3%) has a plain explanation: RL needs no teacher data and can explore tasks the teacher never solved.
Keep the framing straight, though. The Harness VLA line improves the harness and memory; this line improves the model. The paper does not test stacking both, and that is the obvious follow-up.
There is no dedicated limitations section; the caveats sit in the text. Comparisons with memory-equipped planners are labeled system-level references: the teacher trajectories were collected with memory, so the paper cannot split how much of the SFT gain is distilled memory content versus planning skill. Read the 71.0% vs 72.1% near-match with that in mind.
CaP-X numbers stay low in absolute terms, and the Goal and Spatial suites barely move, because the teacher succeeded on only 26 of 60 tasks (5 of 20 Goal, 2 of 20 Spatial), leaving SFT with little data and RL unable to compensate. All results live in the LIBERO simulation family, with no real-robot experiments. The gap to Claude Code (Opus 4.7) with memory remains above 11 points. RL ran on 2,304 episodes, a modest budget, and whether more data raises the ceiling is untested. Code and checkpoints were listed as forthcoming at submission.