StreamPI: Streaming Multimodal Temporal Modeling for Vision-Language-Action Models
Zhe Liu, Jinghua Hou, Yuxiang Lu, Zhenya Yang, Xianzhe Fan, Junwei Luo, Junyi Li, Ruihua Han, Zhi Hou, Hengshuang Zhao
cs.CV
2026-08-27
StreamPI binds each frame to the instruction, streams via KV cache, and trains with random intervals on π0.5 with no extra parameters, lifting cup-in-sleeve from 60% to 92%.
Strong VLAs such as π0 and π0.5 still run single-frame: current image in, action out, no history. That design blocks two things. Memory: after a cup is covered and shuffled, the current frame does not say where the target went. Spatial precision: sliding a pen into a narrow bottle neck, or a paper cup into a sleeve, needs motion parallax and cross-frame structure that one image does not give.
A window of frames restores context, but sequence length grows linearly and real-time control suffers. A separate video encoder also sits off-distribution relative to a pretrained VLA. StreamPI, from the University of Hong Kong and ACE Robotics, adds temporal reasoning to π0.5 without any new parameters.
The core is instruction-anchored temporal modeling. At each timestep the multi-view images and the language instruction form one atomic unit. Inside the unit, attention is bidirectional so vision and language mix. Across units, attention is causal, so the model only sees the past. The instruction is reattached to every frame and is not diluted by a growing pile of visual tokens.
Inference uses a KV cache: past Key/Value tensors stay in memory, the new frame is encoded once, then attends to the cache. Extra sequence length is absorbed by the LLM backbone's length extrapolation. The attention pattern is entirely a mask change; weights are inherited from π0.5. Training uses T = 3 or 5 frames.
The second piece is random-interval streaming training. Real robots deliver observations asynchronously. A policy trained on a fixed stride becomes brittle on the robot. During training the inter-frame interval is drawn from [3, 7], and the earliest frames in the window are randomly masked to mimic a cache that is not yet full. A larger interval, for example inferring every 3 frames, also leaves wall-clock time for action execution.
Four real-robot tasks on AgileX PiperX dual arms with three cameras, 100 teleoperated demos each.
| Task | π0.5 | StreamPI | Delta |
| Shell game (15 trials) | 46.7% | 80.0% | +33.3 |
| Rolling bottle (30) | 26.7% | 63.3% | +36.6 |
| Pen into narrow bottle (30) | 40.0% | 66.7% | +26.7 |
| Cup into sleeve (25) | 60.0% | 92.0% | +32.0 |
LIBERO is already crowded. π0.5 averages 96.9; StreamPI at T=5 reaches 98.3. LIBERO-Long moves 92.4 to 95.0, LIBERO-Goal 96.8 to 99.6, LIBERO-Spatial stays 98.8. On CALVIN, average completed sequence length is 4.547 against 4.313 for π0.5 and 4.090 for MemoryVLA; success at step 5 is 85.0% vs 79.5% vs 69.4%.
Ablations: switching intra-pair attention to causal drops LIBERO-Long by 5.6 points at T=5. Random intervals versus a fixed δ=1 lift T=5 average from 97.0 to 98.3. A model trained at T=5 almost holds at T=3 and still beats single-frame π0.5 at T=1.
On an RTX 4090, single-frame latency is 94.4 ms; T=5 is 103.6 ms, 9.2 ms more.
A strong single-frame VLA does not need a new module to grow a memory. Rewrite the attention mask, pin the instruction to every frame, and jitter the sampling interval. That recipe covers both memory tasks and fine insertion, with near-constant latency, which is more deployable than concatenating a window.
The useful setting is specific: tasks that need cross-frame memory, or geometry that one frame cannot resolve. LIBERO-Spatial does not move. Static spatial relations are already in one image, and extra temporal tokens can just inject irrelevant motion.
There is no standalone limitations section. Real-robot samples are small: 15 shell-game trials, 30 rolling-bottle trials, 10 per insertion pattern, and no confidence intervals. The robot comparison is almost only π0.5; there is no head-to-head with a windowed multi-frame VLA on the same hardware for both success and latency. The KV cache is flushed when it exceeds T, so memory is finite. LIBERO is near saturation, so the 1.4-point average gain is mostly evidence that the method does not break a strong baseline. Random-interval training helps in simulation; the paper does not isolate asynchronous onboard timing as its own control, so the robustness claim is mostly a design argument.