ActiveScale: Scaling Active Perception for Robots across Model, Data, and Hardware
Shuai Zhou, Kaisheng Pang, Wenxuan Song, Wenjie Zhang, Xinhu Zheng, Haoang Li
cs.RO, cs.LG
2026-09-16
CMU and HKUST-GZ add pose-grounded camera tokens to π0.5, mid-train on 1,000 hours of ego and robot data, and raise success on five real active-perception tasks from 30% to 70%.
A VLA with a fixed camera cannot see into a backpack, a drawer, or the underside of a table. People move their heads. Most VLAs neither estimate the current camera pose nor put "where to look next" in the same action space as the arms.
Prior active-perception systems split camera training from arm training (SAPAVE), or pretrain on roughly 200 hours of egocentric data (EgoMI). Three gaps remain. The model has no pose-grounded way to bind views of one scene. The data recipe does not treat human head motion as supervision at the 1,000-hour scale. The hardware is usually a static tabletop, so mobile manipulation is out of reach.
The paper ships a model, a data recipe, and a platform together.
The backbone is π0.5: a PaliGemma VLM plus a flow-matching action expert. The front camera is a four-frame clip at 16-frame strides (t−48, t−32, t−16, now). A learnable camera token follows each front-view visual token. Its hidden state feeds a VGGT-style 9-D camera head (translation, quaternion, two fields of view) with component-wise ℓ1 losses. The head is dropped at action inference. History uses block-causal attention; wrist, language, and action tokens may attend to every valid camera block. Actions cover both arms and the active camera.
Training starts from pretrained π0.5. Human–robot mid-training mixes sources 1:1. Human clips come from indoor household subsets of EgoLive and EgoVerse; robot data come from AgiBot World, RoboCOIN, and the authors' platform, totaling over 1,000 hours, described as more than 5× prior active-perception efforts. Human head and wrist trajectories are expressed in the camera frame of the earliest valid observation in the window. Task-specific post-training then uses AMP teleoperation and keeps only flow matching plus the camera loss.
AMP is a modified AgileX Cobot-Magic: a differential-drive base, two Piper arms for bimanual work, and a third Piper arm carrying the active camera. One operator drives viewpoint, both arms, and the base from a Meta Quest 2. State and actions live in a robot-centric frame that moves with the base, 23 dimensions. Each task family gets 150 post-training demonstrations.
Five real tasks, 20 rollouts each, with success requiring every stage. The baseline is π0.5 post-trained on the same demos. Recent active-perception VLAs did not release full recipes, so they are not compared head-to-head.
| Task | π0.5 success | ActiveScale |
| Bottle from backpack | 40% | 75% |
| Object from drawer | 35% | 55% |
| Object under table | 10% | 85% |
| Vegetable into pot | 15% | 50% |
| Object into instructed box | 50% | 85% |
| Mean | 30.0% | 70.0% |
Mean task progress rises from 41.6% to 78.4%. The under-table gap is the clearest: π0.5 often never starts a grasp, while ActiveScale looks down first. Ablations: dropping mid-training cuts mean success from 70% to 62%. History without camera tokens can hurt the backpack task relative to a single-frame baseline; adding pose supervision recovers all five families. On an RTX 4090, 50-step action chunks run at 264.5 Hz, enough for 30 Hz control with async execution. Mobile manipulation is shown only as a single-operator teleop demo, with no learned-policy success rate.
Active perception is split into three coupled constraints: the model must pin observations from moving cameras onto one scene, the data must supervise looking, and the hardware must let one person collect coordinated demos. Camera tokens are a light touch, with no point cloud or depth injected into the VLA. Teams already on π0.5 can reuse the mid-training recipe. The cost is equally clear: 1,000 hours of mixed training and a custom three-arm platform are not casual to copy.
There is no standalone limitations section. The only learned baseline is π0.5, because peer active-perception VLAs lack public weights, so the SOTA claim sits on a narrow comparison. Twenty rollouts and 150 post-training demos per family are tight. The camera head exists only in training; at inference, cross-view binding lives entirely in the token representation, and that representation is not probed on its own. Learned mobile active perception is left as future work. Human data are filtered to indoor household clips, so transfer depends on that filter. Per-task success rates are read from bar charts, and the mid-training gain on the boxing task is unstable across figures.