SSAD 2026 lecture lays out state representation and interaction challenges for AD world models

At the SSAD 2026 workshop on world models for autonomous driving, @abursuc delivered a series of talks systematically laying out the key considerations for building autonomous driving world models, covering architecture, state representation design, and interactivity.

Confirmed

Most world models today follow a general architecture: a tokenizer compresses inputs and produces state representations, while a world model module takes the current state and conditions to output the next state. The author gave special credit to Wayve's Alex Kendall, whose team was among the first to lay out this thinking for the community.
The core disagreement across approaches lies in what the tokenizer should encode and how it should be trained: options include frozen self-supervised visual features, a focus on semantics, geometry, or task relevance/predictability, or drawing rich information from DINO distillation.
State representations need to faithfully reconstruct pixels so the model can run internally, but should avoid over-focusing on high-frequency content.
The state must accommodate information streams at different frequencies—for example, the world model state needs to be updated with new actions at every 10Hz—making how to carry and update multi-frequency information a central challenge.
When intervening in the environment, there can be correlations between post-intervention states and actions, raising two key questions: how to enable and integrate interactivity within the state space, and how to represent the action space.

Why it matters

This synthesis unifies design choices scattered across papers under one main axis—what the tokenizer encodes and how it is trained—providing a clear framework for comparing and researching autonomous driving world models; meanwhile, interaction problems such as multi-frequency information updating and action space representation remain open challenges on the path to truly interactive world models.

2026-09-17 ~ 2026-09-17 · 7 related posts

Primary sources

Building world models for autonomous driving: state representation trade-offs — abursuc · 2026-09-17 16:12 (UTC+08:00)
World models share one architecture — the tokenizer is where methods diverge — abursuc · 2026-09-17 16:01 (UTC+08:00)
Key considerations for building autonomous driving world models — abursuc · 2026-09-17 16:13 (UTC+08:00)

Anatomy of modern world models: a tokenizer compresses states, a module predicts the next — abursuc · 2026-09-17
[source] World models share one architecture — the tokenizer is where methods diverge — abursuc · 2026-09-17
SSAD 2026 talk maps the zoo of state representation designs for world models — abursuc · 2026-09-17
[source] Building world models for autonomous driving: state representation trade-offs — abursuc · 2026-09-17
[source] Key considerations for building autonomous driving world models — abursuc · 2026-09-17
Course thread: interventions get hard when states and actions are correlated — abursuc · 2026-09-17
World model interactivity: handling mixed-frequency state updates at 10Hz — abursuc · 2026-09-17