SimWAM: A Simple World Action Model for End-to-End Autonomous Driving
Zongchuang Zhao, Xin Zhou, Tianyang Xu, Zhengyang Sun, Kaixuan Zhou, Honglin Li, Dingkang Liang, Xiang Bai
cs.CV
2026-08-08
Co-trained video and action experts with an isolated attention mask let the video branch be dropped at inference; SimWAM reaches 91.5 PDMS on NAVSIM and zero-shots nuScenes.
World-Action Models (WAMs) try to help end-to-end driving by treating generated future frames as "imagination" that guides action. The catch is that most WAM planners still have to actually generate those future frames at inference time, following an imagine-then-act recipe, and video diffusion is too slow to be comfortable on a car.
Two experts are co-trained: a video expert initialized from the pretrained video generator Wan2.2-5B, and an action expert that is a lightweight DiT (hidden size 1024). They are trained with joint flow matching (rectified flow models both the trajectory and the future frames). The key design is an "isolated attention mask": the current observation latents are visible to both future-frame tokens and action tokens, but the future-frame tokens and action tokens cannot see each other. The action expert therefore learns only from the shared observation representation and never depends on future frames.
After training, the video DiT and the future-frame decoder are simply discarded, leaving an action expert that predicts trajectories directly with no future generation at inference. The two experts share no parameters and interact only through attention, so each can swap its backbone and tune its scale independently.
A reinforcement-learning stage is stacked on top, optimizing the compositional NAVSIM PDM reward with group-relative advantages, applied only as rank-32 LoRA on the action expert's attention projections, and trained only on hard scenes where imitation PDMS is below 90.
| Setup | PDMS |
| Action-only | 86.6 |
| joint video training | 90.3 |
| RL | 91.5 |
| Human reference | 94.8 |
On NAVSIM navtest with a single front camera, SimWAM reaches 91.5 PDMS, ahead of SGDrive (91.1), ExploreVLA (90.4), DriveWAM (90.1), and DriveLaW (89.1). Swapping the video backbone is also stable: Wan2.1-1.3B, Wan2.2-5B, and Cosmos-Predict2.5 give 90.2, 90.3, and 90.4 PDMS, so the method is not tied to one video model. It zero-shot transfers to nuScenes with the lowest average collision rate (0.04%) and a competitive average L2 of 0.96 m. Inference latency is about 518 ms at 384x672 resolution with 10 sampling steps on an A100, since only the action expert runs.
The headline is not the SOTA itself but "train with world modeling, infer without imagination." As video generation keeps improving, SimWAM treats it as a swappable training-time signal: swapping in a backbone that was pretrained on driving video (Cosmos-Predict2.5 reaches 90.4) is free upside with no change in deployment cost.
NAVSIM is a non-reactive benchmark where other cars do not respond to the ego car, so the score is not a proxy for real road tests. It uses only a single front camera, which limits field of view. The RL stage runs only on a hard subset and adds a modest 1.2 PDMS. The 91.5 still leaves a 3.3-point gap to the 94.8 human reference.