Qwen-Drive-1.0: An Initial Step towards a Vision-Language Foundation Model for Autonomous Driving
Xin Zhou, Zongchuang Zhao, Zhibo Yang, Mingsheng Li, Humen Zhong, Shuai Bai, Du Chu, Ruizhe Chen, Zhaohai Li, Jun Tang, Qiuyue Wang, Mingkun Yang, Jiazhao Zhang, Dayiheng Liu, Dingkang Liang, Xiang Bai
cs.CV
2026-09-01
Qwen-Drive-1.0 adds a BEV head and planner to Qwen3.5-4B, reaching 43.95 nuScenes mAP and 90.7 NAVSIM PDMS, with general VLM average 66.41 versus 67.40.
Most driving VLAs adapt a general vision-language model by continued scene QA. Fluent captions do not constrain 3D layout, depth, or occupancy. Heavy domain adaptation also forgets the general skills a cockpit still needs. On a shared vehicle SoC, two models throw away the point of cockpit-driving integration.
Qwen-Drive-1.0, from the Qwen team and HUST, leaves the pretrained VLM architecture unchanged and hangs two modules off it: a BEV perception head that emits inspectable 3D detection, semantic occupancy, and BEV map segmentation, and a Planning Expert that flow-matches future ego trajectories.
A shared vision encoder and Qwen3.5-4B take multi-view, multi-frame inputs. View and frame tags from the ordinary vocabulary mark camera and time. QA uses frame-major order, planning uses view-major order. The BEV head reads shallow encoder features and post-VLM semantic features, lifts the former with a predicted depth distribution, fuses them in a BEV transformer, and decodes three tasks. Head-only training is not enough, so stage 2 updates the encoder and VLM as well, mixed with general vision-language data.
The Planning Expert is a 32-layer, 1.1B diffusion transformer conditioned on cached K/V from the VLM's eight grouped-query softmax layers. It predicts 50 waypoints over 5 s at 10 Hz. Stage 3 trains only the expert. Stage 4 turns the last three Euler steps into a stochastic policy and applies group-relative advantages, with PDMS, Rater Feedback Score, and a shared displacement term as rewards. Planning data is about 2.83M samples from NAVSIM, OpenScene, WOD-E2E, and PAI-AV; 24.2% include a planning-reasoning trace.
Perception is scored on a remapped seven-class label space. On nuScenes, Qwen-Drive-1.0-SFT reaches 43.95 mAP, 42.83 NDS, and 60.99 map mIoU, beating same-encoder BEVFormerV2, BEVFormerV2, and PETRv2 by 2.01 mAP, 3.05 NDS, and 3.37 map mIoU. Occupancy is not the best result: 19.82 Occ mIoU on nuScenes versus 25.72 for BEVFormerV2. The paper points at OpenScene label noise and offline completion.
Driving QA averages 69.43 versus 63.52 for Qwen3.5-4B, a 5.91 point lift. LingoQA moves from 70.40 to 77.80, SURDS gains 24.9% relative, Ego3D RMSE drops 40.9% to 7.78. General knowledge-group average is 66.41 versus 67.40 for the base, under one point. Spatial-group average is 53.96 versus 52.99. Driving specialists such as Alpamayo-1.5-10B sit at 14.65 on the general group.
| Planning setup | Metric | SFT w/ reasoning | RL |
| NAVSIM PDMS | higher better | 88.2 | 90.7 |
| WOD-E2E test RFS | higher better | 7.78 | 7.91 |
| AlpaSim off-road | lower better | 24% | 12% |
On WOD-E2E val, RL cuts 5 s ADE from 2.31 to 1.27; on test, ADE barely moves while RFS still rises. On the leakage-free PAI-AV subset, 3 s average ADE is 0.42 m versus 0.36 m for Alpamayo-1.5, and candidates are tight (minADE 0.39 vs 0.17).
This is closer to an engineering case for a shared cockpit-driving model than to a new planner. The VLM skeleton stays, 3D geometry is read from an external head, and general ability drops by less than a point. The planning lift comes from a unified waypoint format plus reward finetuning: SFT already matches imitation methods, and RL mainly halves closed-loop off-road rather than carving more open-loop ADE.
Occupancy lags dedicated multi-task detectors, so the "3D probe" is really a detection-and-map probe. Planning traces mis-identify the governing cause when timescales mix: a red light 20 m ahead wants early braking, a child 5 m ahead wants an immediate stop, and the emitted trajectory may ignore the stated immediate cause. After RL, AlpaSim all-event close encounters are still 41%, with an AlpaSim score of 0.16 versus 0.36 for Alpamayo-R1. PAI-AV candidates lack diversity. Public driving QA was rewritten and filtered by Qwen3.5-Plus at a 55.9% keep rate, which can inflate in-distribution scores.