DeepMind's FlipTest benchmark: best video world models score below random guessing
AlexGDimakis · x · 2026-11-12
Romy Luo (Google DeepMind internship), Alex Dimakis and colleagues introduce FlipTest, a benchmark testing whether video world models actually learn the state of the world.
Core idea: minimal pairs. Each task uses two nearly identical starting frames on opposite sides of a physical tipping point with opposite ground-truth outcomes — a book stack leaning on a prop falls when the prop is yanked away but stays put otherwise; a block shifted 1mm right falls off the table while 1mm left it stays. A true world model must flip its prediction across the pair.
Key findings:
- Tasks span simulated and real-world scenarios
- Current video world models mostly fail: even the best model scored below random guessing overall
- Bigger models, newer checkpoints, physics-informed pretraining, and reasoning tuning didn't fix it
Since these models are increasingly used as world models for robotics, planning, and autonomous driving, the authors argue they haven't learned the underlying world state that determines outcomes. Paper and code are coming soon; a blog post and video are available.
More from Research
- Microsoft Research's ATLAS & JANUS: Neural Samplers for Metallic Glass Materials Discovery — LesterMackey · 2026-11-12
- Steering Qwen3-32B to feel "small" makes it more prone to social engineering — Sauers_ · 2026-11-12
- Why is there no pretrain-postrain co-design? Companies ship their org charts — SonglinYang4 · 2026-11-12
- Google Research trains medical models that understand patient context with clinicians — burny_tech · 2026-11-12
- Every Curve Is a Compressed Argument: Key Data Science Plots Explained — burny_tech · 2026-11-12
- AI formalizes Fermat's Last Theorem in 30 hours, Yang-Mills construction wins Nobel laureate committee sign-off — rand_longevity · 2026-11-12