Reinforcement Learning for Real-Time Vision-Language-Action Policies
Perry Dong, Kuo-Han Hung, Dorsa Sadigh, Chelsea Finn
cs.RO, cs.LG
2026-09-16
Stanford's Real-Time EXPO-FT pairs a slow VLA with a fast edit policy. On four dynamic robot tasks and 10 minutes of online data, average success rose from 42% to 97% (29/30).
Large VLAs are slow. A robot running at 30 Hz cannot wait: by the time the model emits an action, the observation it used is already stale. That mismatch is a distribution shift, and on dynamic tasks success collapses.
Prior real-time schemes such as RTC (Real-Time Chunking) inpaint the already-committed prefix and start computing the next action chunk while the current one is still executing. They keep control smooth under latency. They are mostly imitation learning, so they have no path past the demonstration distribution. Dropping RL on top of RTC still predicts from old observations, so the reliability that RL is supposed to buy never fully shows up.
The Stanford group extends their own EXPO-FT into Real-Time EXPO-FT by splitting action generation across two timescales.
The slow path: a pretrained VLA asynchronously samples several candidate action chunks from a delayed observation, with the in-flight prefix inpainted as in Training-Time RTC so consecutive chunks stay coherent.
The fast path: a lightweight edit policy, conditioned on the latest observation, applies a bounded residual to the remaining actions of each candidate. A Q-function then picks the highest-value chunk among the raw and edited candidates and executes it immediately.
Edits are clamped to a small interval, and the Q signal trains only the edit network, not the VLA backbone. The large model stays a behavior prior; the small net reacts to whatever changed during the delay window. Conditioning the edit on the current state keeps the executed policy Markovian, which is what standard RL updates assume.
Training has three losses. The VLA is fit with prefix-conditioned flow-matching behavior cloning: the prefix is treated as clean actions, noise is added only to the postfix, and the loss is masked to the postfix. The edit policy maximizes Q of the edited chunk with an entropy bonus. The critic does a chunk-level TD backup. Sampling the VLA many times per backup is expensive, so a lightweight noise-space critic (from their earlier FASTER work) ranks Gaussian seeds, one seed is decoded, then edits are scored in action space.
On the robot, the base model is π0.5 with LoRA (a gemma2b language backbone and a gemma300m action expert). Control is Cartesian plus gripper velocity at 30 Hz on a DROID single-arm setup, with 224×224 side and wrist images. Simulation does not use a VLA: it fine-tunes a state-based flow-matching policy from the Kinetix / RTC suite under a 4-step delay. Before online RL, the VLA is supervised-finetuned to roughly 30% success on demonstrations. Online training runs without human corrections.
Four dynamic real-robot tasks, online data capped at 10 minutes, 30 evaluation trials each. Native VLA latency is about 67 ms. Ball balancing, object passing, and soccer kicking add 100 ms of extra delay (about 167 ms, d=5). Dynamic picking keeps 67 ms (d=3); extra delay drives non-async baselines near zero.
| Method | Average success |
| SFT | 12.5/30 |
| SFT w/ RTC | 18/30 |
| RLPD | 4.5/30 |
| DSRL w/ RTC | 20/30 |
| EXPO-FT | 18.8/30 |
| EXPO-FT w/ RTC | 25/30 |
| Real-Time EXPO-FT | 29/30 |
Per task: dynamic picking 30/30, object passing 30/30, ball balancing 28/30, soccer kicking 28/30. The last two hit the 10-minute cap; the first two reached 30/30 earlier. RLPD, a lightweight Gaussian policy, scores 0/30 on picking and passing. SFT averages 42%; the new method averages 97%.
On 10 Kinetix environments with a 4-step delay (4 seeds × 100 eval episodes): Real-Time EXPO-FT averages 96.2% and is the best delayed method in every environment. That is 14.5 points above EXPO-FT w/ RTC (81.7%) and 20.1 points above DSRL w/ RTC (76.1%). Zero-delay RLPD averages 81.4%. A delayed policy beat a lightweight RL agent that sees the current state.
Sweeping delay from 0 to 4, RTC drops while this method holds. On real object passing, speeds from 0.5× to 1.25× stay near 100% success; EXPO-FT without real-time edits falls as the object moves faster.
Reliability and latency pull in opposite directions for VLAs: bigger models are better priors and slower to query, which is the wrong trade for fast-changing scenes. The split here is usable in a lab. Let the large model propose chunks asynchronously; let a small net edit against the latest frame. Ten minutes of online data and no human corrections during training (resets are still manual) is a realistic recipe if you already have a pretrained VLA and a success detector.
This is an incremental systems paper, not a new learning paradigm. The ingredients are EXPO plus RTC. What previous work did not finish was stacking RL's reliability gains on top of real-time execution.
The authors list two: a human still resets the scene, and each task needs its own success detector; they did not compare reward designs.
A few more caveats. Simulation uses a state-based flow policy, not a VLA, so the delay results do not automatically transfer to vision-language models. Real-robot eval is 30 trials per task with a single seed and no reported variance. The 10-minute wall-clock cap does not equalize environment steps (about 5k on passing, about 18k on the others). RLPD and DSRL get asynchronous high-UTD updates while EXPO-style methods update per episode; the authors say this favors the baselines. Edit scale is a per-task hyperparameter (0.05 for kicking, 0.1 otherwise). Sparse binary rewards plus rule-based detectors may not survive more open-ended tasks.