Apple-$π$: Benchmarking Thinking with Video Towards Law-Grounded Physical Intelligence
Runmao Yao, Kairui Hu, Yukang Cao, Ruisi Wang, Shulin Tian, Ziang Cao, Weichen Fan, Ziqi Huang, Yuhao Dong, Hao Li, Zhaoxi Chen, Zhongang Cai, Lei Yang, Ziwei Liu
cs.CV
2026-07-18
Apple-pi uses 400 classical-mechanics videos and a Perception, Formulation, Deduction protocol to test whether video models reason from physical law. The best video model, Seedance 2.0, scores only 0.473, exposing a bottleneck where perception succeeds but deduction fails.
Video generation models are now pitched as "world models" that have internalized physical law. But existing benchmarks check only whether the final frames look plausible, not whether the model got there by reasoning through physics. A correct-looking clip might reflect real mechanics, or it might be luck producing something that merely resembles the right outcome. Apple-pi asks the pointed question: when video models reason, do they work from physical laws like Newton, or from intuition like Aristotle? (The "Apple" in the name is Newton's falling apple, not the company.)
Apple-pi has three parts. First, the Orchard dataset: 400 videos covering ten classical-mechanics tasks, sourced from simulation (243 cases), self-recorded real scenes (121) and internet clips (36). Tasks split into a single-law branch for confounder-free diagnosis and a multi-law branch for generalization. The three pillars are universal gravitation, conservation of momentum and Newton's first law, with objects restricted to four primitives: sphere, cube, cylinder, cone.
Second, a three-stage protocol: Perception, Formulation, Deduction. The input is an infographic-annotated first frame; the model generates the video in a chain-of-frames, treating generation as its visible reasoning trace. Perception identifies physical quantities and object locations, Formulation selects the law and predicts target states, Deduction produces a trajectory consistent with the law.
Third, an evaluation suite: MLLM-based subjective scoring for format compliance, plus physics-law-grounded objective measures (Mask IoU, spatiotemporal IoU, velocity accuracy). The combination localizes where a model fails, not just whether it fails.
Eleven models benchmarked, five video-generation and six unified understanding-generation models:
| Type | Best | Score |
| Video gen | Seedance 2.0 | 0.473 |
| Unified | GPT Image 2 | 0.704 |
| Unified | Nano Banana 2 | 0.699 |
Three findings all point the same way: models are far from reliable physics world simulators. First, scores decline from Perception to Formulation to Deduction; perception passes, formulation is marginal, deduction collapses, so success at intermediate steps does not transfer to time-consistent dynamics. Second, multi-law tasks are markedly harder than single-law; models cannot carry physical state across law transitions. Third, a sim-to-real gap appears across all models, with grounding and tracking under realistic visual conditions as the weak spot.
For anyone building video generation or world models, Apple-pi is a diagnostic frame, not a leaderboard. It tells you which stage a model dies at, which is more useful than a single final-frame score. The core takeaway is a measurement warning: the high scores those "world models" post may be propped up by final-frame plausibility rather than genuine physical reasoning.
Physics scope is only classical mechanics; no electromagnetism or fluids. The object vocabulary is four geometric primitives, far from real-world complexity. Evaluation uses a single-camera perspective with a fixed input interface. Objective measures test law-consistency, not overall visual fidelity. Real-video ground truth carries measurement noise, and the protocol assumes temporal normalization across frame rates. Eleven models is a limited sample.