VFlash: Fast, Accurate Visual Planning via Flash Primitive Decoding
Yu-Cheng Chou, Shihao Wang, Jin Huang, Chi-Pin Huang, Chiao-An Yang, Xianzhe Fan, Jing Wang, Zhiqi Li, Yuanguo Kuang, Ruijie Zheng, Johan Bjorck, Sifei Liu, Hang Su, Abhishek Badki, Lei Zhang, Alan Yuille, Linxi "Jim'' Fan, Jan Kautz, Zhiding Yu
cs.RO, cs.CV
2026-11-06
VFlash writes a full trajectory in one pass, without matching autoregressive tokens. 3D ADE drops by up to 61%; MolmoSpaces success goes from 62.6% to 70.2%.
Vision-language models for manipulation and driving often emit a visual plan before the action model runs: a box, a point, or a trajectory over the next few seconds. The format is fixed in advance. A box is four numbers. A trajectory is a fixed count of 2D or 3D coordinates.
Autoregressive decoding emits those coordinates one token at a time. A longer trace, or a joint 2D and 3D trace, adds latency in proportion to the token count. An agent that replans on new observations then has to drop waypoints.
Speculative decoding keeps a draft only while it matches the target model's next token. One quantization bin can be a few centimeters in space and still fail that check. DFlash accepts 5.5 tokens per round on subtask text, and only 2.2 inside trajectory blocks, so decoding time falls by about half.
Flash Primitive Decoding treats one complete geometric object as the parallel unit. The target VLM stays frozen. Its features feed a five-layer, non-causal diffusion drafter that fills the masked block in one forward pass. Block length comes from the primitive type and a tool call, so a ten-point dual-arm 2D+3D trajectory, a single box, and one dense-motion frame each get their own length.
Every slot has a known role. The main experiments use 1,001 special tokens for coordinates normalized to 0-1,000, and inference restricts the vocabulary. If the target writes plain digits, VFlash uses a sign plus five digit slots, then renders that grid back into the original text so a longer number cannot shift later slots.
Subtask text still has to match the target tokens. A visual primitive is committed in full once the schema check passes. Box coordinates that fail the LocateAnything spatial-confidence guard are filled in by constrained autoregressive decoding. Trajectories are gated only by the schema check. Labels are annotated coordinates, not the target's own continuation, and the loss is uniform across the block. One drafter sees both short text blocks and primitives.
The base model is Qwen3-4B with a native-resolution MoonViT encoder and an MLP connector. The 4B target is fine-tuned with next-token loss, then frozen while the drafter trains. Dense motion instead freezes the released MolmoMotion-AR model. A valid primitive costs one drafter pass plus one target pass. That target pass updates the cache and exposes hidden states to the action model. More waypoints do not add sequential steps.
Every comparison keeps the same target and changes the decoder.
On 2D trajectories, all six errors on ShareRobot-Bench and VABench-V drop. Hausdorff distance falls by about 7%, and DFD and RMSE by 3-6%. On VABench-V the errors are about 40% lower than LoHo-Manip, though the 4B autoregressive target already beats most listed models. On ShareRobot, a DFD of 0.194 still trails RoboBrain-2.0 at 0.167.
| Setting | Metric | AR | VFlash |
| ShareRobot / VABench-V | HD | 0.175 / 0.125 | 0.162 / 0.116 |
| AgiBot, 3D only | ADE (m) / within 5 cm | 0.083 / 50.3% | 0.042 / 73.1% |
| AgiBot, 2D+3D | ADE (m) / within 5 cm | 0.105 / 39.8% | 0.041 / 73.9% |
| MolmoSpaces | Average success | 62.6% | 70.2% |
3D trajectories move the most. On InternA1, AgiBot, and MolmoBot, ADE drops 18-49% for 3D-only outputs and 28-61% when 2D and 3D are written together. On AgiBot, waypoints within 5 cm rise from 50.3% to 73.1% in the 3D-only setting. When both streams are required, autoregressive decoding falls to 39.8% while VFlash stays at 73.9%. Joint 2D+3D writing makes the autoregressive 3D ADE 15-51% worse than 3D-only. Wrist-camera images used only in training cut ADE another 3-12% when testing still uses the main camera.
Boxes do not get that accuracy gain. On COCO and Dense200, VFlash with autoregressive fallback matches mean F1 and runs about 2.6-2.8 times faster, with higher F1 at IoU 0.50 and lower F1 at IoU 0.95. Without fallback, Dense200 mean F1 falls from 60.10 to 39.09.
Dense point motion, drafted against frozen MolmoMotion-AR, is about 25 times faster in a preliminary single rollout. DAVIS ADE and FDE fall from 1.227 and 2.108 to 0.893 and 1.716. HOT3D FDE falls 8%. ADE on HOT3D and WorldTrack still trails autoregression, WorldTrack FDE rises from 0.261 to 0.300, and PWT is 7-12% lower.
On MolmoSpaces the policy is a world action model post-trained from Cosmos-Predict2.5. It reads hidden states for the question, the subtask, and a ten-point 2D+3D trajectory. Average success is 70.2%, against 62.6% for the original text encoder. Pick gains 2.5 points, Pick and Place and Close gain 11.6 and 12.8, and Open gains 3.5. The average beats PrimeR0 at 60.3%, while Close at 77.0% still trails PrimeR0 at 81.3%.
The plan ablation changes the reading. With the same VLM, autoregressive 2D+3D plans score 61.0%, under the 62.6% baseline. Autoregressive 3D-only plans reach 69.3%. Replacing trajectory states with image states yields 63.7%. Most of the downstream gain is a 3D plan that stays intact. VFlash is what keeps the joint 2D+3D plan from breaking that 3D trace.
The LIBERO family uses one PIv3 flow-matching head against a frozen Qwen3-VL-4B. The gain grows from 0.8 points on LIBERO (93.4% to 94.2%) to 1.9 on LIBERO-Plus and 6.8 on LIBERO-PRO. On LIBERO-PRO, Spatial, Object, and Goal rise by 11.1, 9.4, and 6.9 points. Against subtask-only conditioning, Spatial, Goal, and Long add 2.6, 1.2, and 2.0.
For driving, the planning VLM stays frozen and a small diffusion head is trained alone. Supervised PDMS on NAVSIM is 87.1, above ReCogDrive at 86.8 and below Qwen-Drive-1.0-SFT at 88.2. Reinforcement learning on the head reaches 90.7, tying Qwen-Drive-1.0-RL. On the Waymo test split, RFS moves from 7.79 to 7.86, 0.05 short of the best listed 7.91.
At batch size 1, MolmoSpaces step latency falls from 3552.7 ms to about 703.3 ms on Jetson Thor and from 3356.5 ms to 649.9 ms on DGX Spark, about 80% lower. The paper reports 5.1-5.2 times on Thor and Spark, and about 3.6 times on H100. NAVSIM is about 3.4-3.6 times on Thor and Spark and 2.6 times on H100. Waymo on H100 is about 4.5 times. The cut is in decoding. On Thor, autoregression takes 75-87% of a step. Growing a driving primitive from 36 to 84 tokens leaves VFlash decoding at 72-78 ms.
Replanning time was spent walking coordinates one by one. On an embedded platform, one primitive is two forwards, and a denser trajectory barely adds latency.
The drafter beats the 4B autoregressive target it reads from, most clearly when 2D and 3D share one trajectory. Autoregression conditions each coordinate on the previous one. VFlash scores the whole layout at once. The target's token distribution is not the ceiling for visual outputs.
MolmoSpaces makes that operational. Autoregressive 2D+3D plans from the same VLM score 61.0%, below 62.6%; VFlash plans reach 70.2%. The action head attends to those hidden states. A bad plan is not repaired downstream. Driving ties Qwen-Drive. The saving is sequential plan decoding, and reinforcement learning never updates the VLM.
The joint gain in speed and accuracy stops at robot trajectories. Boxes get worse at high IoU, and Dense200 mean F1 falls to 39.09 if rejected boxes are kept. PWT is lower on all three point-motion sets, and WorldTrack FDE goes up. Direct acceptance helps only when annotated coordinates are easier to fit than the target's next-token distribution.
The 25 times figure is a preliminary single rollout, separate from the staged timings on Thor, Spark, and H100. The control experiments stay in simulation. LIBERO itself gains 0.8 points; the larger jump is LIBERO-PRO, and that jump is not rechecked with another action head.
Reinforcement learning lifts NAVSIM PDMS to 90.7 with the plans held fixed, so those points belong to the action head. Supervised PDMS at 87.1 still trails Qwen-Drive-1.0-SFT at 88.2. The drafter has five layers, and the target is task-tuned first. Unseen primitive lengths and untuned VLMs are not tested. Each request still returns one primitive.