RL$^2$-VLA: Adaptive RL Latent Compositional Steering with Test-Time Scaling for Vision-Language-Action Models
Derek Ming Siang Tan, Shailesh Shailesh, Srikrishna Iyer, William Wei Jie Teo, Yuanliang Ju, Qiao Gu, Guillaume Sartoretti
cs.RO
2026-07-29
RL² composes a lightweight offline RL policy into a frozen VLA action flow, firing only when a failure detector predicts the base model is about to fail. Out-of-domain success rates climb up to 17.3 points.
Vision-Language-Action (VLA) models map images and language instructions straight to robot actions, and they are the dominant route to general-purpose robots. They do well on tasks seen in training and fall apart on anything out-of-domain. The paper gives blunt numbers: a rephrased instruction drops success from 38.2% to 14.2%, an unseen tabletop environment drops it from 70.2% to 36.0%.
The standard fix is test-time steering: skip retraining, sample several candidate actions at inference, and let a verifier pick the best. But every candidate comes from the same policy, so failure modes correlate and resampling only circles the same hole. Worse, almost every method intervenes at every timestep, injecting noise into steps the base model already handled correctly.
The intuition behind RL² is human. You do not deliberate over alternatives when you reach for a mug on a clear table, but you switch approach when the mug is half hidden. Diversity helps when the default action is about to fail; it is just noise when the default is already right.
RL² wraps three modules around a frozen VLA and never touches its weights.
First, it trains a lightweight offline RL policy. The policy is conditioned on latents pulled from the VLA action expert and predicts an action chunk, trained on the same offline data (BridgeV2, DROID) with each datapoint passed through the VLA once to cache its latent. The training trick is QAM (Q-learning with Adjoint Matching). Backpropagating the critic through flow matching multi-step denoising is numerically unstable, so QAM instead computes a time-dependent guidance signal backward from the terminal action and uses it as a stable first-order target. Tuning a single inverse temperature τ is enough to beat behavior cloning.
Second, compositional steering. The VLA generates actions by flow matching, advancing noise toward valid actions along a velocity field. At each denoising step, RL² takes a weighted average of the VLA velocity and the RL velocity: v = w·vVLA + (1-w)·vRL, with w drawn from N(0.5, 0.25) clipped to [0,1]. Each candidate keeps the behavioral prior from large-scale imitation learning while the RL term pulls it toward high-value actions that demonstration rarely visited. For autoregressive VLAs like OpenVLA that cannot do flow composition, it falls back to a Gaussian perturbation strategy.
Third, steer only on failure, the core claim. The authors first run a scaling-law study: on a BridgeV2 validation split binned by action error into success and failure groups, error falls exponentially with sample count in both, but in the failure group compositional steering drops it fastest, while in the success group almost every method, RL² included, underperforms the Rephrase baseline, with RBF and RL² among the worst. That is why steering has to be adaptive. In practice, RL² attaches a lightweight failure detector called SAFE (an LSTM over VLA latents) that outputs a per-frame [0,1] failure score; conformal prediction sets a per-timestep threshold δt, and composition fires only when the score exceeds it. A verifier (RoboMonkey or CoVer) then picks the best candidate to execute.
Three simulation benchmarks plus a real arm.
| Setting (model / scenario) | Baseline | Gain |
| SIMPLER / OpenVLA (in-domain) | Repeated | avg +7.5%, best +19.4% |
| SIMPLER / π₀ (OOD prompts) | Rephrase | avg +10.1%, best +14.7% |
| SIMPLER / π₀ (OOD environments) | Rephrase | avg +8.5%, best +14.6% |
| PolaRiS / π₀.₅ (OOD prompts) | Rephrase (31.8% to 42.7%) | avg +10.9%, best +17.3% |
| Real PiperX / π₀ (4 OOD tasks) | Rephrase | avg +17.5% |
The PolaRiS table is the cleanest read. π₀.₅ alone scores 14.3% success; adding Rephrase (sampling under VLM-rewritten instructions) lifts it to 31.8%; non-adaptive RL² only reaches 33.8%, barely better than Rephrase; adaptive RL² reaches 42.7%. The jump from non-adaptive to adaptive is the whole argument for gating on failure. On the real arm, adaptive beats non-adaptive RL² by another +14.2%, restating the same point on hardware.
Two ablations are hard. Latents are indispensable: replacing the VLA action-expert latents with raw observations collapses success from 39.3% to 0.5%, because the expert features are far more expressive than a small ResNet encoder. And RL beats BC by up to +4.5% under the same composition framework, confirming that diversity has to come from RL exploration, not imitation. The detector matters too: SAFE beats using CoVer as the detector by up to +3.5%, and beats an always-on trigger by more.
Cost is negligible. On an RTX 5090 at batch 1, the π₀ forward pass costs 232 ms, the CoVer verifier 96 ms, the QAM steering policy 12 ms, and SAFE 1 ms; even at batch 128, QAM is 48 ms and SAFE 2 ms. The two added networks are a rounding error next to the VLA itself.
For robotics practitioners, RL² is a retrain-free, modular, plug-and-play path to OOD robustness. It is agnostic to the VLA (OpenVLA, π₀, π₀.₅ all work) and to the verifier (RoboMonkey, CoVer both work), bolting two lightweight networks onto a frozen base. The broader takeaway is a mechanism, not a trick: intervention has a cost, so spend it only when the model is about to fail. That failure-gated view of generate-candidates-then-pick should transfer to any inference-time selection framework, not just robotics.
Honest framing: these are gains from offline RL trained on simulation data, not a capability breakthrough. RL² does not teach the policy action types it never had; it shifts the existing action distribution toward better candidates.
The authors list three, all fair. They only compared lightweight steering functions, not differentiable steering from large VLMs or other VLAs. The α-selection heuristic for the conformal threshold needs extra test-time evaluation and uses balanced accuracy, which the authors admit a timeliness metric might beat; the detector also still needs online rollout collection to train. And the work assumes a reliable verifier, setting aside what happens when the verifier itself is wrong.
One more concern surfaced while reading. In the real-world experiments the SAFE detector did not transfer from simulation, since it was trained on a small set of sim rollouts, so the authors had to collect fresh real-world rollouts and retrain it. Each new deployment environment may need its own detector recalibration, and the plug-and-play claim bends at the detector. The PolaRiS numbers also rest on 50 trials across 3 seeds per task, with non-trivial variance bands.