PAWBench Reveals Video Generators Fail Probabilistic Alignment Tests

Researchers from Hugging Face and other institutions have released PAWBench and PAWEval, the first systematic evaluation of whether video generation models, acting as stochastic samplers, can match reference behavior distributions — a property known as "probabilistic alignment." The results show that although all 11 tested video generators produce realistic footage, none can accurately capture both "what can happen" and "how likely it is to happen" at the same time.

Confirmed

• The research introduces PAWBench and PAWEval, formalizing the probabilistic alignment requirements for world models and evaluating the physical world modeling capabilities of 11 video generators.
• Experiments show the models fail to match reference behavior distributions: even when explicitly told the expected outcome, models produce it only 38–58% of the time.
• Model behavior shows systematic biases: they overreact to irrelevant cues while underreacting to genuine physical changes.
• The authors tried three fixes — clearer outcome instructions, more diverse sampling, and additional training — each addressing part of the problem, but none providing an end-to-end repair.
• The authors illustrate the issue with a dice-rolling example: a model may generate realistic-looking footage yet repeatedly show only a few outcomes, failing to cover the true distribution of results.

Why it matters

• Author @RisingSayak notes that producing one good video may suffice for a creative tool, but falls far short for a world model used in planning — it must capture both "which outcomes are possible" and "how likely each outcome is." The current models' shortcomings in this respect mean there remains a fundamental gap before they become usable world models.

2026-08-28 ~ 2026-08-28 · 5 related posts

• [source] PAWBench Study: Current World Models Fail to Match Reference Behavior Distributions — Yuandong Pu · 2026-08-28
• PAWBench: Does a video model capture not only what can happen, but how often? — RisingSayak · 2026-08-28
• Why World Models Need "Probabilistic Alignment": One Good Video Isn't Enough — RisingSayak · 2026-08-28
• [source] PAWBench Results: Even Told the Target Outcome, Video Models Hit It Only 38–58% of the Time — RisingSayak · 2026-08-28
• [source] PAWBench: Video Generators Fail Probabilistic Alignment for World Modeling — RisingSayak · 2026-08-28