AD-E2E-JEPA: A Joint-Embedding Predictive Architecture For End-to-End Autonomous Driving
Haoran Zhu, Wancong Zhang, Yann LeCun, Anna Choromanska
cs.RO, cs.AI, cs.CV, cs.LG
2026-09-28
AD-E2E-JEPA compresses JEPA planning 100x with a SIGReg projector. No policy: 72.9 EPDMS and 2.8 m error at 8192 candidates on NAVSIMv2; IL rises 80.2 to 85.4.
End-to-end driving still mostly clones human trajectories from a few camera frames. The policy is reactive. It never builds an explicit model of how the scene moves. Human logs are noisy, and they are not always the right target.
JEPA (Joint-Embedding Predictive Architecture: predict the future in latent space, skip pixel reconstruction) tries another route. Learn an action-conditioned world model, roll out candidate trajectories, pick the one whose predicted state sits closest to a goal. Off-the-shelf JEPA world models split on cost versus accuracy. LeWM plans on a global CLS token in about 0.7 s per scene on an A100, but scores 48.3 EPDMS (NAVSIMv2's planning score with multiplicative safety gates) on a 100-scene slice, with 12.4 m final displacement. DINO-WM and JEPA-WM keep dense encoder patches, reach 68.3 and 74.2 EPDMS with about 4 m error, then spend 91.8 s and 101.0 s per scene. A full 12,146-scene sweep would take more than ten days. The cheap model cannot plan. The accurate ones cannot run.
AD-E2E-JEPA, from NYU and AMI Labs, freezes DINOv3 ViT-L and trains a projector plus an action-conditioned predictor on the compressed tokens.
Only the front camera is used. Context is 4 frames (2 s at 2 Hz). The future horizon is 8 frames (4 s). The action is the relative pose step (Δx, Δy, Δθ). Human trajectories condition the world model. They do not supervise a driving policy.
The projector is two stride-2 convolutions, applied to history and future encoder maps. Planning patches drop 16x. Width drops from 1024 to 256. An AdaLN predictor with RoPE does one-step prediction in that space, trained with MSE. Targets get a stop-gradient. SIGReg then pushes each patch, at each timestep, toward an isotropic Gaussian so the projector cannot collapse distinct DINOv3 vectors to a constant. An optional rollout loss adds autoregressive consistency over all 8 future frames, with truncated BPTT so gradients do not flow through earlier steps.
Planning skips CEM (cross-entropy method, the usual iterative sampler over action sequences). Too expensive here. Candidates come from VADv2's clustered vocabulary of 8,192 driving trajectories, subsampled to 256 by angle unless stated otherwise. The goal is the image F frames ahead; evaluation uses the ground-truth frame. Each candidate is rolled out. The winner is the one whose predicted latent is nearest the goal latent. No policy is trained.
For transfer, the predictor is dropped. DINOv3 and the projector feed a cross-attention trajectory head trained with MSE. The control is the same head with a randomly initialized projector.
Default search uses 256 candidates on one A100, with the ground-truth future frame as the goal. EPDMS† drops the safety gates and keeps the quality average.
The 100-scene subset exists so DINO-WM and JEPA-WM can be compared at all:
| Method | EPDMS | EPDMS† | Time | FDE | Top-1/5 hit |
| LeWM | 48.3 | 73.9 | 0.7 s | 12.4 m | 6/18 |
| DINO-WM | 68.3 | 91.4 | 91.8 s | 3.9 m | 40/73 |
| JEPA-WM | 74.2 | 90.9 | 101.0 s | 4.0 m | 45/75 |
| AD-E2E-JEPA | 76.6 | 92.4 | 0.8 s | 4.2 m | 27/59 |
| +rollout (trainval) | 72.3 | 91.9 | 0.8 s | 3.2 m | 47/78 |
The single-step navtrain model posts the best subset EPDMS (76.6), but the share of scenes where the ground-truth trajectory ranks in the cheapest top-k lags the dense-patch baselines. Rollout training plus the 70-hour trainval split (navtrain is 10 hours) is what catches geodesic error and that hit rate.
Full 12,146 scenes, trainval with rollout:
| Candidates | EPDMS | EPDMS† | FDE | Δθ | Time |
| 256 | 67.3 | 84.1 | 4.0 m | 3.5° | 0.8 s |
| 8192 | 72.9 | 86.5 | 2.8 m | 2.0° | 18.2 s |
Against LeWM's 39.8 EPDMS on the same full split, navtrain AD-E2E-JEPA is 23.7 points higher. Goals sit about 20 m away on average; displacement is 4.0 m at 256 candidates and 2.8 m at 8,192. Hit rate falls as the vocabulary densifies: 53.8%/82.7% top-1/5 at 256, 15.3%/33.8% at 8,192. Ranking the exact human path is harder when many nearby paths exist.
Imitation learning, single front view, no perception labels: a random projector scores 80.2 EPDMS, the pretrained one 85.4. WA-JEPA reports 91.7, but that run uses multi-view cameras and a 64+32 A800 two-stage schedule. The paper lists it as a reference, not a matched baseline.
The engineering takeaway is narrow and usable. JEPA planning cost lives in dense patches. Two stride-2 convs plus SIGReg cut about 100 s down to 0.8 s, and planning quality stays in the same band.
The projector also moves. Dropped into a plain imitation head it adds 5.2 EPDMS. Teams that do not want closed-loop search can still treat it as a representation compressor.
Driving without a policy holds only under a privileged protocol: the goal is a future ground-truth image. A car does not get that frame. The result measures whether actions and future latents line up, not whether this is a deployable planner.
Relative to WA-JEPA-style imitation stacks that buy 91.7 with a pile of GPUs, this is a lighter complementary line.
There is no Limitations section.
Zero-shot planning aims at the ground-truth camera frame F steps ahead. That isolates world-model quality from policy learning, and it is also the largest evaluation gift. No experiment replaces that goal with a command, a map, or a predicted image.
Search is retrieval over 8,192 clustered VADv2 trajectories, not continuous control. 0.8 s (256) and 18.2 s (8,192) are A100 numbers. They can fit a 2 Hz NAVSIM tick at the small vocabulary; they do not fit higher-rate control.
Safety gates are the weak end. On the full split at 256 candidates, drivable-area compliance (DAC) is 82.0 and extended comfort (EC) is 35.5. The same imitation head reaches 93.7 and 88.7. EPDMS multiplies collision, drivable area, driving direction, and traffic-light terms, and the world model is not trained on those. The extra EPDMS† score exists for that reason.
On the 100-scene slice, adding rollout to navtrain drops EPDMS from 76.6 to 70.4 while final displacement improves. The paper points to subset variance, and to a mismatch between latent-distance training and NAVSIM driving scores. Rollout helps on the full set, but the two evaluation slices do not rank models the same way. Comparisons with DINO-WM on 100 scenes should be read with that gap in mind.
No closed-loop vehicle results, no extra cameras, no LiDAR. There is also no zero-shot bake-off against Drive-JEPA or WA-JEPA under one protocol.