SSAD 2026: Rethinking AV Benchmarks and the Case for Open Source
At #ssad2026, Valeo AI researcher Kashyap Chitta reviewed the evolution of end-to-end autonomous driving evaluation and argued for open source, making evaluation and open source two key themes for accelerating progress in the field.
Confirmed
- He recapped how known problems with nuPlan's open-loop evaluation (see "Is Ego Status All You Need for Open-Loop Autonomous Driving") led to the rise of NAVSIM.
- Using software-industry analogies such as Linux vs. Windows NT, Chitta argued that open source is the right path for AI, recommending Bill Gurley's writing.
- Bill Gurley published a new piece, "From Open Source Software to Ope...", on open-source strategy, tied to the question of whether the automotive field can see an open-source wave like LLMs.
- A researcher asked whether the automotive industry can replicate the LLM open-source wave and its rapid progress, listing current players including Valeo.ai and noting room for more entrants, while sharing concerns about current autonomous driving deployment realities.
- Chitta also recalled the shift in ML evaluation culture: in the 1990s almost no papers included benchmarks, while by the 2020s papers without benchmarks are nearly unpublishable; building common test frameworks was hard, and the progress is impressive.
Why it matters
- The shift from open-loop evaluation to benchmarks like NAVSIM directly shapes the direction and credibility of end-to-end autonomous driving research.
- An LLM-style open-source wave in automotive could lower barriers and accelerate iteration, though deployment realities suggest a gap between ideal and practice.
2026-09-16 ~ 2026-09-16 · 5 related posts
Primary sources
- E2E driving evaluation shifts: nuPlan open-loop flaws exposed, NAVSIM benchmarks drove two years of progress — abursuc ·
- Kashyap Chitta argues open source is the way, using the Linux vs Windows NT analogy — abursuc ·
- Bill Gurley on Open Source Strategy: can automotive AI replicate the LLM open-source wave — abursuc ·
- Researcher calls for an open-source wave in autonomous driving, citing current deployment reality check — abursuc · 2026-09-16
- [source] Bill Gurley on Open Source Strategy: can automotive AI replicate the LLM open-source wave — abursuc · 2026-09-16
- [source] Kashyap Chitta argues open source is the way, using the Linux vs Windows NT analogy — abursuc · 2026-09-16
- From no benchmarks in the 1990s to no papers without them: a short history of ML evaluation — abursuc · 2026-09-16
- [source] E2E driving evaluation shifts: nuPlan open-loop flaws exposed, NAVSIM benchmarks drove two years of progress — abursuc · 2026-09-16