Deferred Exposure of Future Trajectories for Verifiable Reasoning in Autonomous Driving VLMs
Zixuan Huang, Yang Zhou, Kaixuan Wang, Guli Zhang, Hongyan Xie, Yakun Zhu, Hao Geng, Yikun Ban, Deqing Wang
cs.AI
2026-08-03
Teacher VLMs shown the GT trajectory rationalize: severe hallucinations rise 29% to 50%. A verifiable MCQ with deferred candidates hits 77.9% accuracy, general vision intact.
Most autonomous-driving systems that put a large vision-language model (VLM) in the loop use it for perception and reasoning, then hand off to a smaller action expert for the actual trajectory. To make the VLM's chain-of-thought (CoT) more reliable, the standard recipe has a teacher model produce the training reasoning traces, and that teacher is usually shown the ground-truth future trajectory from the driving log.
This paper isolates a side effect nobody had pinned down. The moment the teacher sees the correct trajectory, it works backward from the answer instead of forward from the scene. The authors call this trajectory anchoring bias, after the anchoring effect in cognitive psychology. They ran a human-scored controlled study: with the GT trajectory hidden, severe hallucinations in the CoT sat at 29%; with it revealed, they jumped to 50%, and human pairwise preference collapsed from 60.5% to 24%. The damage concentrated in precisely the causally hard scenes where honest reasoning matters most.
That is a real problem. The model lands on the right action for the wrong reasons, and training on such traces teaches rationalization dressed up as reasoning.
The fix has two layers.
The first layer, AD-MCQ, addresses verifiability. Generating a continuous trajectory is hard to score and tangles high-level decisions with precise geometry, so the authors recast planning as multiple choice. K-means over 489K logged trajectories yields a codebook of 8192 discrete prototypes (think 8192 canonical ways to drive). For each scene, the GT trajectory maps to its nearest prototype, and similarity-retrieved distractors fill out the options. Picking the right candidate becomes an exactly checkable decision, with no coordinates to generate.
But showing candidates up front lets the model lazily compare options rather than reason about the scene, which just swaps a GT anchor for a candidate-set anchor. The second layer, DEFT, splits inference into two turns. Turn one gives the model only the scene and forces a committed high-level decision before any candidate is visible. Turn two reveals the candidates and asks the model to match its already-committed decision to the closest trajectory. The decision comes first; the trajectory only verifies.
Training uses GRPO (group relative policy optimization, from DeepSeek-R1), with a reward equal to MCQ correctness times a rubric score. A VLM generates scene-specific rubric criteria offline (which scene facts the reasoning should cite), and an online text-only judge checks how many criteria the turn-one trace hits. The GT trajectory never enters turn-one reasoning, never feeds the rubric generator, and the judge sees neither the image nor the options, only the text. That yields a hard verifiable signal without letting the answer leak back into the reasoning.
The main numbers are on Qwen3-VL-8B and Qwen3.5-4B over AD-MCQ-500, 500 causally hard scenes.
| Setting (Qwen3-VL-8B) | Accuracy | CFS |
| Base, candidates shown, no training | 28.1% | — |
| DEFT, two-turn inference, no training | 56.6% | 0.431 |
| Candidates visible + same RLVR reward | 61.1% | 0.428 |
| DEFT-RLVR, full method | 77.9% | 0.658 |
Just hiding the candidates and splitting into two turns (no training) lifts accuracy from 28.1% to 56.6%. Under an identical reward, deferring candidates beats showing them by 15.3 points, direct evidence that anchoring is dragging performance down. With rubric supervision added, DEFT-RLVR reaches 77.9% and pushes causal faithfulness from 0.431 to 0.658.
General-vision capability barely moves, and that matters: the average over 12 vision-language benchmarks edges up from 54.81% to 56.09%. Distillation from a teacher, by contrast, drops it to 49.80%. Reinforcement learning only nudges token probabilities for the model's own samples rather than yanking the whole distribution, which is why it preserves general skill. On the out-of-distribution nuScenes set, accuracy rises from 39.6% to 49.5%, a sign the model learned transferable scene-to-decision reasoning rather than source-specific shortcuts. The rubric adds only 0.5% training overhead.
For anyone building end-to-end driving stacks, the main contribution is methodological. Turning planning into a verifiable multiple-choice task sidesteps the hardest part of reinforcement learning for driving: getting a reward you can actually check. No closed-loop simulator, no action expert, and no dynamics model are needed; the VLM alone can run RLVR, and difficulty is tunable through distractor construction. That makes it a usable experimental platform for the field.
For anyone training reasoning models, the anchoring-bias finding stands on its own as a warning. Do not let the teacher that writes your CoT data see the answer first, or a large share of the reasoning you collect will be post-hoc rationalization that fails exactly when the scene is hard. The lesson generalizes well beyond driving to any dataset built as "give the answer, then ask for the explanation."
The paper has no dedicated limitations section; the following are the gaps that stood out.
First, every evaluation is open-loop log replay on Waymo and nuScenes. There is no closed-loop driving and no on-road validation, and picking the right candidate is still a long way from executing it safely.
Second, the MCQ framing is a proxy. Trajectories are quantized into 8192 prototypes with a reconstruction error floor of 0.279 m ADE, so trajectory fidelity is capped by the codebook. Real vehicle control needs continuous output, which this scheme does not directly provide.
Third, the two core quality metrics, CFS and HLD, are scored by a 397-billion-parameter model rather than ground truth, so judge bias is possible. The authors report agreement with human annotators in an appendix, but it remains a model-graded metric.
Fourth, the anchoring-bias claim rests on a 100-scene human study, a modest sample. And the work trains only the VLM reasoning head of the VLA stack; the downstream action expert and the full end-to-end pipeline are not validated here.