E2E driving evaluation shifts: nuPlan open-loop flaws exposed, NAVSIM benchmarks drove two years of progress
abursuc · x · 2026-09-16
- At the #ssad2026 workshop, researchers reviewed the evolution of end-to-end autonomous driving evaluation: nuPlan's open-loop protocol has known flaws, and the paper "Is Ego Status All You Need for Open-Loop Autonomous Driving" exposed how ego status can shortcut the benchmark.
- NAVSIM then emerged as a lightweight-simulation benchmark suite, acting as a super-enabler for e2e driving model progress over the last two years, though not without imperfections.
- Broader observation: in the 1990s no papers had benchmarks; in the 2020s no papers ship without them.
Related event: SSAD 2026: Rethinking AV Benchmarks and the Case for Open Source(5 posts)→
More from Research
- Radial gate objective cuts diffusion transformer FID from 16.24 to 11.83 — Junha Park · 2026-11-02
- REA: async brain-cerebellum runtime for robots where policies never wait on the LLM — Pengfei Zhou · 2026-11-02
- Microsoft's MemSteer turns unlabeled user histories into agent memory supervision — MicrosoftResearch · 2026-11-02
- SciGame benchmark uses counterfactual worlds to expose LLM agents' priors vs evidence — Yusheng Zhao · 2026-11-02
- On Terence Tao's blog, Frank Calegari calls on mathematicians to adapt after OpenAI's 700-paper release — Hesamation · 2026-11-02
- Tsinghua & Microsoft's HarnessMM cuts agent token use by up to 87.93% while raising success rate — rohanpaul_ai · 2026-11-02