Hidden future trajectories make autonomous-driving VLMs reason more faithfully

Buaa1 · hf · 2026-08-04

Driving VLMs get more faithful reasoning when future trajectories are hidden until after the decision

This paper studies a failure mode in autonomous-driving VLA/VLM training: annotation pipelines often expose the teacher model to the ground-truth future trajectory, which creates trajectory anchoring bias.

Main finding

When the future trajectory is visible up front, the teacher tends to rationalize the already-revealed outcome instead of inferring a decision from scene evidence. That leads to less causally faithful chain-of-thought and more hallucinations, especially in difficult scenes.

Proposed fix

Remove the ground-truth trajectory from the reasoning prompt.
Introduce AD-MCQ (Autonomous-Driving Multiple-Choice Question), which turns planning into selecting among explicit trajectory candidates.
Propose DEFT-RLVR (Deferred Exposure of Future Trajectories for RLVR), which moves future trajectories from pre-decision anchors to post-decision verification targets.

Results

DEFT-RLVR improves autonomous-driving reasoning.
It preserves or even improves general visual capabilities.
AD-MCQ gives controllable difficulty through candidate construction and works with VLM-only inference.

Why it matters

The paper argues that future-trajectory supervision should be used for verification, not as a shortcut that leaks the answer before reasoning starts.

Original post →

Hidden future trajectories make autonomous-driving VLMs reason more faithfully

More from Apps