Beyond Data Scaling: Representation-Centric Continued Pre-training for Vision-Language-Action Models
Senqiao Yang, Chengyao Wang, Yuxin Chen, Zixuan Wang, Longxiang Tang, Haokun Gui, Jinhui Ye, Changsheng Lu, Xiaoyang Wu, Mingkang Zhu, Pengguang Chen, Shu Liu, Zhuotao Tian, Hengshuang Zhao, Bei Yu, Jiaya Jia
cs.RO, cs.CV
2026-08-28
VLAct continues pre-training a 4B VLM on public robot data with frozen shallow layers, multi-head continuous co-supervision, and a partially unified action layout. It hits 82.6% on LIBERO-Plus and 92.5% on RoboTwin 2.0 Clean, and with 20% of unseen GR-1 data reaches 49.5% versus 47.6% for full-data GR00T-N1.6.
Robot trajectories are not scrapable like image-text. Collection is expensive and coverage of scenes, objects, tasks, and embodiments stays sparse. If continued pre-training only fits actions on that sparse set, three failures show up: the vision-language prior washes out on narrow scenes; a single action head locks backbone geometry to its decoder; per-embodiment heads refuse to share even gripper open/close semantics.
VLAct treats the scarce resource as representation quality under a fixed trajectory budget. At fine-tuning time the action head is freshly initialized. Backbone weights are the only changing part.
The run starts from Qwen3-VL-4B and continues pre-training on public sets including DROID, InternA1, RoboCoin, and MolmoAct, on 16 GPUs. Three ingredients shape the backbone and are discarded for downstream work.
The vision-language prior is protected by freezing the vision encoder and the lower half of the LLM, updating only upper layers and heads, and mixing captions, which ablations found to be the strongest anchor. Shallow-layer protection is worth about 3.7 points on LIBERO-Plus and 3.4 on Agilex. Fine-tuning unfreezes everything.
Action features are co-supervised by three continuous heads: OFT, PI, and GR00T. A pilot showed FAST discrete tokens transfer but lose amplitude; OFT-only pre-training is strong for OFT fine-tuning and drops 5.4 points on PI and 22.3 on GR00T. Multi-head training is there to stop decoder lock-in.
Cross-embodiment sharing uses one head and a partially unified layout: gripper dimensions are shared, incompatible arm joints stay embodiment-specific and masked when inactive. Periodic joints use a wrap-aware loss modulo 360 degrees, so 179 and -179 are 2 degrees apart, not 358.
LIBERO-Plus trains on standard LIBERO and tests on perturbations:
| Method | Camera | Robot | Noise | Layout | Total |
| π0 | 13.8 | 6.0 | 79.0 | 68.8 | 53.6 |
| Abot-M0 | 60.4 | 67.9 | 86.4 | 82.6 | 80.5 |
| Qwen3VL-OFT | 47.0 | 60.1 | 73.1 | 79.2 | 75.0 |
| VLAct | 73.9 | 68.4 | 86.0 | 83.3 | 82.6 |
That is 7.6 points over the same-backbone OFT fine-tune and 2.1 over Alibaba's Abot-M0, with Camera moving from 47.0 to 73.9.
On RoboTwin 2.0 Data Scaling, VLAct-OFT reaches 92.5% Clean and 90.8% Random; the PI head hits 93.0% Clean. In the Base setting with 50 clean trajectories per task, VLAct-OFT gets 80.5% / 41.5% versus 61.7% / 10.5% for Qwen3VL-OFT. On a real Franka, short-horizon in-domain success is 92.5% versus 77.5%, dual-arm 72.0% versus 44.0%, and full object-substitution long-horizon still 83.3% versus 46.6%.
Continued pre-training saw only Franka and AgileX. On held-out GR-1, 20% of downstream data already yields 49.5%, 50% yields 51.0%, and full data 54.0%, against 47.6% for full-data GR00T-N1.6, 48.8% for the same backbone without this recipe, and 37.0% for π0.5. On RoboDojo, VLAct ranks sixth of 35 by success (7.60%) with score 10.66, above every named world-action model. Memory is a hole at 0.66 / 0.56%.
Matched fine-tuning protocols isolate the backbone. Gains of 7.6 to 21.4 points sit on the representation recipe, not on inheriting a stronger head. For groups without proprietary trajectories and with about 16 GPUs, this is a public-data path that can sit next to industrial VLAs.
The paper does not argue against scale. It argues that how limited trajectories shape the backbone is a separate axis. Real-robot and unseen-embodiment transfer carry more information than another saturated sim board.
Only 4B was trained. Larger VLMs may want different freeze depths and mix ratios. RoboDojo Memory is near zero, so the recipe does not buy episode memory. The leaderboard does not normalize compute, and the top rows are mostly industry submissions. Pre-training heads are thrown away, so a new embodiment still needs a chosen head and downstream data; 20% is not zero-shot. Real-robot cells use 10 trials.