VLA reinforcement learning updates are low-rank and concentrate in overlooked Timestep Modules

HOLILAB · hf · 2026-10-01

HOLILAB systematically studies how RL reshapes vision-language-action (VLA) policies in parameter space.

Across flow-based VLAs (π₀.₅, GR00T N1.5/N1.6) on LIBERO, ManiSkill, MetaWorld, and CALVIN, RL induces substantially lower-rank updates, highly concentrated in the action expert's Timestep Modules — a small, previously overlooked component that captures a disproportionate share of RL's gains.
RL specializes these modules to the discrete denoising timesteps used during rollouts; this discrete-timestep training underlies the low-rank updates.
Among module outputs, the shift vector changes most distinctly under RL: shift update directions predict task success (ROC-AUC up to 99.6%), and their geometry correlates with cross-task transfer patterns.
Steering along shift update directions further improves RL-trained policies without additional RL training.

Implications for more efficient, interpretable VLA post-training.

More from Embodied