PAWBench: How Far Are We from Probabilistically Aligned World Modeling?
Yuandong Pu, Le Zhuo, Sayak Paul, Gabriel Jorge Menezes, Avram Đorđević, Shiyang Li, Yifan Zhou, Bin Fu, Wenlong Zhang, Junjun He, Yu Qiao, Yihao Liu, Jingbo Xing, Xi Chen
cs.CV, cs.AI
2026-08-28
PAWBench tests eleven video generators as samplers of stochastic physics across 50 scenes; none jointly match reference odds and recover the valid outcome support.
Video generators are increasingly sold as world models: give an initial frame and an action prompt, get a continuation. One plausible clip is not a model of a stochastic process. A coin toss, a spinner, a bowling ball can all end in several valid ways from the same start. Current benchmarks mostly score a single video for fidelity, temporal coherence, and whether physics looks right. They rarely ask whether repeated samples collapse onto one outcome, or whether the frequencies match.
This paper calls the missing requirement probabilistic alignment: under a fixed observation and action, the induced outcome distribution should cover the valid support and put mass in the right places.
PAWBench has 50 scenes in eight mechanism groups. PAW-Calibration covers 25 scenes with analytic or symmetry-derived reference distributions, such as coin tosses and wheels. PAW-Coverage covers 25 scenes whose valid terminals can be listed but whose probabilities cannot, such as bowling and bottle flips. Each scene freezes the source image and action prompt and draws K=50 rollouts per model.
PAWEval uses rubric-guided Gemini 3.5 Flash to map readable, in-schema videos to terminal labels, and records readout failures separately. Calibration uses conditional total variation distance (TVD); Coverage uses valid-support recovery. A scene passes the readout gate if at least 20 of 50 rollouts are readable. Source images are generated; humans review each scene for visible physical stochasticity, an atomic action, and distinguishable terminals.
After the main board, the paper probes three knobs: language (VLM future sampling, GPT-5.5 prompt engineering, Oracle-specified targets), coupled initial noise (C2C), and LoRA fine-tunes trained on different left/right pencil-fall mixtures.
None of the eleven systems jointly calibrates probability mass, recovers the valid support, and stays readable across most scenes.
| Model | Calibration TVD↓ | Cal. SPR | Coverage↑ | Cov. SPR |
| Cosmos 3 Super I2V | 20.5 | 80% | 55.2% | 92% |
| MiniMax H3 | 24.2 | 68% | 48.7% | 92% |
| Seedance 2 | 30.5 | 100% | 50.9% | 84% |
| LTX-2.3 | 30.1 | 24% | 71.7% | 72% |
| HappyHorse | 43.1 | 92% | 47.1% | 100% |
The broadest-coverage models are not the best-calibrated. Observed mean TVD is 31.2; matched draws from the references stay below 9.22 in 99% of simulations, so finite sampling does not explain the gap. On 888 videos with a clear terminal, PAWEval agrees with the human majority on 722 (81.3%).
Causal probes show incomplete or wrong shifts when the physical transition changes, and probability mass moving when only distractor text changes. Language is weaker still: the best VLM future sampler, GLM-5V Turbo, has Calibration TVD 34.8. Feeding GPT-5.5’s chosen outcomes to four generators raises TVD in every case. Oracle targets improve TVD and coverage, yet generators hit the requested outcome only 37.6%–58.1% of the time. C2C coupled noise explores a 50-rollout gallery more broadly; it does not change the learned distribution. LoRA training that varies the left-fall share from 0% to 100% moves both the upright and left-leaning pencil scenes in the same direction. No checkpoint matches 50/50 on the upright pencil and 100/0 on the left-leaning one.
Looking like a world model and sampling futures at the right rates are different jobs. Planning, risk, and interaction need the second. PAWBench turns that into a score, and shows that prompts, noise, and coarse fine-tuning do not teach a scene-conditioned distribution. Anyone training video world models should treat repeated-rollout outcome histograms as a first-class metric.
The authors list three. Evaluation uses terminal labels, not trajectory-level dynamics. Estimates use a finite rollout budget: more samples can uncover extra outcomes without fixing biased mass. Scenes are controlled and visually parseable, not long-horizon or embodied. The source images are themselves generated, and reference odds rest on symmetry; a real coin need not be 50/50. PAWEval still disagrees with humans on about 19% of clear cases, and failed readouts are dropped from conditional metrics, so TVD averages for low-SPR models such as LTX-2.3 (24% Calibration pass) should be read with care.