From Pixels to Actions--Towards Native Vision-Language-Action Policies
Weichen Fan, Haiwen Diao, Penghao Wu, Siyuan Hu, Ziqi Huang, Tianqi Liu, Mingze Sun, Ziyi Wang, Zhongang Cai, Hanming Deng, Quan Wang, Lei Yang, Lewei Lu, Fangzhou Hong, Zhaoxi Chen, Dahua Lin, Ziwei Liu
cs.CV
2026-11-05
NEO-VLA drops the vision encoder and runs pixels through a 2B backbone. Frozen, its LIBERO average is 90.9%, above Qwen3-VL at 79.5% and Gemma-3 at 84.5%.
A standard vision-language-action model takes images and instructions in and emits robot actions. The image first goes through a pretrained encoder such as CLIP, SigLIP, or DINOv2. The language backbone and the action head see only that fixed endpoint. Grasping and placement depend on local geometry, edges, depth, and small state changes, the signals this compression is most likely to drop. Switching which language-backbone layer is read still sits downstream of that endpoint.
The vision module is already a known weak point: it is sensitive to pruning, and action fine-tuning can damage pretrained visual features. NEO-VLA changes the interface so the action head can read visual computation inside the backbone, before it collapses to one semantic vector.
There is no separate vision encoder, image codebook, or autoencoder. A two-stage convolution with stride 32 emits one token per 32 by 32 pixels into a 40-layer, 2B language backbone. Images keep their aspect ratio, with at most 88 tokens per view. Native RoPE splits position into time, height, and width. Text attention is causal; image tokens inside one visual block attend in both directions, so geometry and semantics are stages of a single stack.
A narrow, randomly initialized expert predicts continuous actions with flow matching. Noise and the target are mixed over time, the network fits the velocity, and inference integrates with 10 Euler steps. The prefix key/value cache stays fixed, and classifier-free guidance is off. Each expert layer reads only its mapped backbone layer. skip-28 takes 28 layers spread across all 40; end-28 reads layers 12 through 39. Those two experts match in width and depth, so the contrast is readout depth. first-28 reads only the front and also shortens the backbone. full-40 uses a deeper expert and is only a higher-capacity reference.
The prefix carries up to three views, camera geometry, control frequency, proprioception, and the instruction. Six state tokens cover the previous and current values of both arms and the body; missing parts are masked. One query predicts a 30-step action chunk. Each arm has 26 channels (7 joints, 3 translation increments, 3 rotation increments, 1 gripper, 12 hand joints); the body has 8 channels over the same 30 steps. Four velocity groups are weighted 2, 1, 1, 1, with end-effector plus gripper heaviest.
Pretraining draws on nine public datasets: 2,231,583 episodes and 9,927.7 hours. Simulated InternData-A1 is 42.42% of stored rows and 15% of sampled rows. A final quality pass removes 3 of 2,231,586 episodes.
Training then has three stages: expert-only for 113,400 steps at batch 8192 and learning rate 2×10^{-4}; 75,000 full-model steps with the backbone at 5×10^{-5}, understanding data on every other step, and an auxiliary next-token loss weighted 0.1; then 75,000 further steps on in-house action data. The controlled comparison does not use that full recipe. Each backbone is pretrained for 75k iterations on Open X-Embodiment under the same setting, then paired with the same action expert.
On that matched setup, LIBERO is scored at step 30,000 with 500 episodes per suite. The NEO row uses the skip-28 readout.
| Backbone | Adaptation | Average success |
| NEO-VLA 2B | expert only | 90.9% |
| Qwen3-VL 2B | expert only | 79.5% |
| Gemma-3 4B | expert only | 84.5% |
| NEO-VLA 2B | full fine-tune | 95.3% |
| Qwen3-VL 2B | full fine-tune | 90.8% |
| Gemma-3 4B | full fine-tune | 91.0% |
Frozen 2B NEO reaches 90.9%, level with the other two models after full fine-tuning. On LIBERO-Long it scores 85.4% frozen, above Qwen full fine-tuning at 78.6% and Gemma at 82.8%. Full fine-tuning lifts NEO to 95.3% and shrinks its lead over Qwen from 11.4 points to 4.5. Gemma's backbone is about twice as large and still behind.
Wiring 13 Qwen3-VL expert layers to intermediate vision-tower features adds no parameters. Frozen success drops from 79.5% to 64.9%. After full fine-tuning the original interface scores 90.8% and the mid-layer interface 90.9%. Raw access to encoder mid-layers does not reproduce the frozen transfer.
Layer maps use the same setup and 2,000 rollouts each. With the 40-layer backbone kept, frozen skip-28 scores 90.9% and end-28 scores 86.7%. After fine-tuning the pair is 95.3% versus 92.5%, and on Long it is 92.0% versus 84.2%. Frozen first-28 reaches 94.9%, but the backbone is shorter, so that number does not show that earlier is always better. Frozen first-14 versus last-14 is 93.6% versus 89.6%, and depth changes there too.
Replacing one attachment's cached keys and values with another observation moves translation error most at layers 0-10. Magnitude-matched noise stays within 0.004 cm. Wrist-image sensitivity peaks at layers 6-10, and instruction sensitivity peaks at layer 0. Wrist-camera pose shows up at layer 0 only when actions are predicted in that camera frame.
Action-only post-training stops the model from producing valid text, and RoboTwin 2.0 clean-to-randomized success falls to 10.7%. Mixing understanding data on alternate steps moves MMBench-EN from 82.7 to 81.2, keeps the other four understanding scores within 1.6 points, and reaches 69.4% under randomization.
Fully trained public leaderboards do not share one protocol. NEO-VLA averages 97.4% on LIBERO (98.4, 99.4, 95.6, 96.2 across the four suites; 2,000 episodes each at step 27,500). OpenVLA scores 76.5, EO-1 98.4, and MMaDA-VLA 98.0. Qwen-RobotManip reports only an aggregate 99.1. In-distribution scores barely separate the leaders.
LIBERO-Plus is task-weighted. NEO-VLA scores 89.7, highest among models with public code and weights: camera 94.5, language 91.7, noise 96.9, light 96.2, background 96.6, robot 72.6, layout 82.4. π0.5 totals 84.4, with camera at 78.4. π0 totals 53.6, with camera at 13.8. Closed Qwen-RobotManip totals 89.0 and scores 87.3 on layout, above NEO.
On LIBERO-Pro, NEO-VLA scores 98.0 on semantic shifts, 84.3 on object appearance, 3.0 on position, and 6.3 on task changes, averaging 47.9. π0.5 averages 53.3, with position at 20.8. Wording and appearance hold up. Moved objects and new goals do not.
On the official 2.5K-episode RoboTwin 2.0 split, NEO-VLA scores 75.78 clean and 69.40 clean-to-random, a drop of about 6.4 points. π0.5 falls from 70.70 to 46.00, and FastWAM from 77.80 to 1.90. Among releases with public code and weights, NEO has the best randomized score. Its clean score still trails closed Qwen-RobotManip at 89.58.
The real tasks are placing a tennis ball on a can, putting tape in a bowl then the ball in the tape, and standing a horizontal can up inside the tape roll. Each model is fine-tuned on 50 demonstrations and tested for 20 trials. NEO-VLA scores 12/20, 17/20, and 13/20; π0.5 scores 10/20, 14/20, and 13/20; StarVLA scores 1/20, 2/20, and 0/20. In-domain and out-of-domain trials are not split. On one RTX 3090 in bf16, 10 Euler steps take 282.5 ms compiled and 286.2 ms until an executable plan. Two-step sampling lowers the plan time to 178.5 ms. Reported success uses 10 steps.
With the expert and the OXE corpus matched, a frozen 2B encoder-free backbone already matches encoder-based models after full fine-tuning. Another fine-tune adds about 4 points, but the representation is usable before that rewrite.
Weights, code, and the action pretraining data are public. Early layers matter more while the backbone is frozen; after full fine-tuning, skip-28 is the better map. Fifty real demonstrations match or beat π0.5. The in-distribution 97.4% sits with the strongest public policies. The gap that remains is camera change, language change, and scene randomization.
The paper keeps the causal claim narrow. Pretraining and capacity differ, so the gain is not only from deleting the encoder. Cross-paper tables also mix simulators, cameras, and data. Appearance robustness is not layout or goal robustness, and that gap is left open.
Frozen first-28 at 94.9% also shortens the backbone. The Qwen mid-layer test uses one 13-layer wiring. Twenty real trials make 12/20 versus 10/20 a fragile margin, and StarVLA's near misses look like an unmatched post-training setup. The 12 hand-joint channels, the loss weights, and the 15% simulated draw are not separated from the architecture.