The Low-Rank Structure of VLA Reinforcement Learning
Minjae Oh, Yoonah Park, Jongwon Lim, Yohan Jo
cs.LG
2026-09-28
RL updates in π0.5 and GR00T VLAs are low-rank and concentrated in the 27%-parameter Timestep Modules; keeping only their updates loses almost nothing; test-time steering adds 2-4 points.
Flow-based VLAs such as π0.5 and GR00T are increasingly post-trained with RL in simulation, following the recipe that worked for LLMs. For LLMs, parameter-level studies showed that RL touches a sparse, low-rank set of weights, and those findings reshaped training practice, LoRA-style efficient post-training included. Nothing equivalent existed for VLA RL, so where the learned signal actually lands in the weights was an open question.
The gap has a real cost. Teams running VLA RL with LoRA pick target modules by LLM convention, attention and FFN projections, with no evidence the gain lives there. The answer from this Seoul National University team: most of the gain lands in a small, previously overlooked component of the action expert called the Timestep Modules, which mainstream LoRA configs (RLinf's default included) omit entirely.
The authors diff BC and RL checkpoints against their base weights across π0.5, GR00T N1.5/N1.6, and SmolVLA on LIBERO, ManiSkill, MetaWorld, and CALVIN, measuring update density (fraction of weights changing by more than 1e-5) and effective rank (fewest singular directions covering 95% of the update's energy).
The component that lights up is the Timestep Modules: the parts of the action expert that turn the denoising timestep τ into scale, shift, and gate vectors. In π0.5 that is a Time MLP plus AdaRMS, 27.58% of the action expert's parameters. These vectors depend only on τ, so with K denoising steps the module emits K fixed sets of modulation vectors, independent of image, instruction, or task progress. Three experiments carry the paper:
Core replacement numbers (success rate %, LIBERO averaged over three suites):
| Setting | Base | Full RL | Timestep Modules only | MLP+Attn only |
| π0.5 LIBERO | 87.7 | 96.5 | 95.1 | 90.5 |
| π0.5 ManiSkill | 42.2 | 89.1 | 87.5 | 49.4 |
| π0.5 MetaWorld | 42.8 | 69.2 | 69.6 | 54.6 |
| π0.5 CALVIN | 61.8 | 87.7 | 87.0 | 62.7 |
| GR00T N1.6 LIBERO-Spatial | 75.6 | 85.0 | 84.8 | 84.6 |
| GR00T N1.5 LIBERO | 54.4 | 90.3 | 61.3 | 57.9 |
On π0.5, keeping only the Timestep Modules' RL updates (27.58% of the action expert) stays within 0.7-1.6 points of the full RL policy on three of four benchmarks and is 0.4 higher on MetaWorld; keeping everything except them drops ManiSkill by 39.7 points, with six-setting average deficits of 5.4 versus 19.7. The rank numbers are stark: under RL the Timestep Modules have effective rank 6-30 versus 250-650 under BC, an order of magnitude, while attention and MLP stay high-rank. Reconstructing the Timestep Module updates from only the top four singular directions retains nearly all performance; deleting those four directions drops five of six benchmarks below the base policy.
The cause is how training samples timesteps: RL specializes AdaRMS to the discrete rollout timesteps, and discrete-timestep BC reproduces the same low-rank structure, so discrete-timestep training itself is the driver. RL-trained image and video models (FLUX with DanceGRPO, Wan2.2 with VideoRLVR) do not show this; their timestep-module updates need 29-80% of available rank. The effect is specific to action policies.
Among the three outputs, scale and gate are mostly rescaled along their original directions while shift picks up new directions. Shift update directions probe episode outcomes at up to 99.6% ROC-AUC (four of five benchmarks above 98.6 on the base policy, versus roughly 50% for random-label controls), and they transfer to the RL policy. Pairwise alignment of shift updates correlates with cross-task transfer (mean task-wise Spearman 0.504, up to 0.794). Steering the RL policy along these directions at test time adds 2.0-4.0 points on all five benchmarks (LIBERO-Spatial 86.7 to 90.0, Goal 90.7 to 94.7), beating matched-norm random directions; steering the base policy recovers 46-90% of the RL gain. LoRA targeted at the Timestep Modules converges faster and higher than the standard attention+FFN target, 93.0 versus 91.4 at step 100 on LIBERO-Spatial.
Anyone running VLA RL with LoRA should revisit the target list; adding the Timestep Modules is close to free, and replacement, SVD, and targeted-LoRA evidence all point the same way.
The conceptual finding carries more weight: in this setting, most of what RL learns can be written as a small set of timestep-specific modulation vectors, independent of observations and instructions. That echoes results in LLM RL where small subnetworks suffice, and it gives a parameter-level account of why residual RL and lightweight adapters over frozen VLAs work. The probing and steering results hand you a cheap outcome predictor and a test-time knob. This is an analysis paper with no new SOTA, but structural findings of this kind tend to outlast leaderboard entries.
GR00T N1.5 is the clear exception: Timestep-Modules-only falls from 90.3 to 61.3, a 29-point loss, and the paper does not explain why; the analyzed checkpoints were retrained in-house after a floating-point mismatch in the official ones. On N1.6 both replacement directions stay within half a point, suggesting a redundancy the paper leaves unexplained.
Everything runs in simulation with PPO and short training (150-320 interaction steps); real robots, longer runs, and other algorithms are untested. The 2-4 point steering gains require a probe-selected sublayer-timestep pair and a per-benchmark steering coefficient, so the setup cost is higher than it looks. The high-rank attention/MLP updates still carry some gain on their own (90.5 versus base 87.7 on π0.5 LIBERO); what they encode is left open.