WCM: A World Critic Model for Vision-Language-Action Reinforcement Learning
Senyu Fei, Xiaopeng Yu, Siyin Wang, Xianzhong Zhao, Jingjing Gong, Xipeng Qiu
cs.RO, cs.CL, cs.CV
2026-08-01
WCM adds a future-prediction head to the robot-RL critic so it learns temporal dynamics; pi-0 hits 84.4 on ManiSkill and OpenVLA-OFT jumps 0.8 to 98.7 zero-shot.
Reinforcement-learning post-training of Vision-Language-Action (VLA) models is now a standard route for robotic manipulation, and RL needs a critic to judge how good each step is. The trouble is that today's critics look at only the current frame, or at a single-frame latent from the VLM backbone. Robotic manipulation is partially observable: whether the hand has touched the object, how far contact has progressed, and what happens next all live in the time series, which a single frame cannot show. Naively stacking frames does not work either, because regressing a scalar return is too weak a supervision to force a cross-temporal representation. WCM fixes exactly this critic.
WCM sits on a lightweight LeJEPA-style architecture. Its core idea is to make the critic predict the future while it estimates value, so that temporal dynamics get baked into the representation.
The key choice is joint optimization. Instead of pretraining a world model and bolting a critic onto it, prediction and value estimation share one backbone, which forces it to actually learn how the environment evolves over time rather than regress a scalar. WCM works on-policy (PPO for autoregressive models like OpenVLA-OFT; Flow-SDE for flow-matching models like pi-0 and pi-0.5) and off-policy (AWR and RECAP respectively, with a unified buffer mixing SFT data and rollouts).
On ManiSkill (IND and OOD settings; delta is gain over the SFT baseline):
| Backbone | Method | IND | OOD |
| pi-0 | SFT | 38.4 | 18.1 |
| pi-0 | +FlowSDE | 78.8 | 39.3 |
| pi-0 | +WCM | 84.4 | 51.5 |
| pi-0.5 | +WCM | 91.9 | 64.4 |
| OpenVLA-OFT | +PPO | 97.7 | 77.1 |
| OpenVLA-OFT | +WCM | 99.0 | 77.9 |
The most striking line is zero-shot transfer for OpenVLA-OFT: without any SFT, adding WCM lifts IND from 0.8 to 98.7. On LIBERO-Plus generalization, pi-0 starting from One-SFT (a single demo per task, 32.3%) reaches 72.8% with WCM, beating Full-SFT (50 demos) at 71.2%; all three backbones show similar gains of +33.7 to +44.7 points. On a real WidowX-250S robot, WCM beats its off-policy baselines (AWR and RECAP) across seven tasks, for example pi-0.5 plus WCM picking up a carrot 44/50 against 33/50.
For anyone doing robot RL, WCM is a concrete and cheap lever: a critic need not be single-frame, and a future-prediction head lets it ingest temporal information, with both on- and off-policy training and both autoregressive and flow-matching VLAs supported. The LIBERO-Plus result that one demo plus WCM beats 50-demo Full-SFT shows it pulls up sample efficiency. The underlying idea, making a value representation carry a world-modeling duty, transfers cleanly to other critic-based RL settings.
The authors list several. History length K=3 is optimal and longer brings little (their reading: three frames implicitly capture second-order dynamics, that is acceleration, while two capture only first-order, velocity). The lambda that trades prediction against value is sensitive on OOD settings (a 10.6-point swing versus 2.7 on IND) and needs careful tuning. Real-world training needs hundreds to thousands of trajectories; 16,800 simulation SFT samples transfer to the real robot at 0% success, so the sim-to-real gap persists. One concern from reading: the zero-shot 0.8 to 98.7 jump is enormous and worth confirming beyond the ManiSkill distribution.