PAWBench Finds Video Generators Fail Probabilistic Alignment as World Models
Researchers from Hugging Face and others released PAWBench and PAWEval, the first systematic benchmark asking whether video generation models, used as stochastic samplers, can match reference behavior distributions — a property the authors call probabilistic alignment. Across 11 tested video generators, none could accurately capture both what may happen and how likely each outcome is, revealing a fundamental gap for world-model use.
Confirmed
- PAWBench and PAWEval formalize the probabilistic alignment requirement for world models and evaluate 11 video generators' physical world modeling.
- Even when explicitly told the desired outcome, models generated it only 38–58% of the time, failing to match reference behavior distributions.
- Models show systematic biases: overreacting to irrelevant cues while underreacting to genuine physical changes.
- Three fixes — clearer outcome instructions, more diverse sampling, and additional training — each addressed part of the problem, but none provided an end-to-end solution.
- Using dice rolls as an example, the authors show models can produce realistic footage while repeatedly sampling only a few outcomes instead of covering the true distribution.
Why it matters
- As author @RisingSayak argues, generating a good video may suffice for creative tools, but world models used for planning must also capture which outcomes are possible and how likely each is — a requirement current models fundamentally fail to meet.
2026-08-28 ~ 2026-08-28 · 5 related posts
Primary sources
- [source] PAWBench Study: Current World Models Fail to Match Reference Behavior Distributions — Yuandong Pu · 2026-08-28
- [source] PAWBench: Does a video model capture not only what can happen, but how often? — RisingSayak · 2026-08-28
- Why World Models Need "Probabilistic Alignment": One Good Video Isn't Enough — RisingSayak · 2026-08-28
- PAWBench Results: Even Told the Target Outcome, Video Models Hit It Only 38–58% of the Time — RisingSayak · 2026-08-28
- [source] PAWBench: Video Generators Fail Probabilistic Alignment for World Modeling — RisingSayak · 2026-08-28