VeriPhy: Agentic Physical Reasoning for World Model Evaluation and Refinement
Wenzhuo Xu, Yuchen Zhu, Chongjian Ge, Xuan Shen, Jing Shi, Jason Kuen, Yongxin Chen, Molei Tao, Christopher McComb, Noelia Grande Gutiérrez, Jiuxiang Gu
cs.CV
2026-09-03
VeriPhy compiles a prompt into typed physical checks and measures the clip with frozen tools. On 149 core clips: 228 of 304 flaws found, vs 164 for question decomposition.
A generated clip can look fluent and still break physics. A scalar quality score cannot say which prompt obligation failed or in which frames. Existing evaluators split the job: rule-level checks, temporal localization, measurements in controlled scenes, audiovisual consistency. What is missing is one chain that compiles those pieces and lets every verdict be traced to evidence.
Open prompts usually ship no rubric. The evaluator has to compile "what to check" from the text, then measure it on pixels, rather than ticking a list of pre-written phenomena.
VeriPhy splits planning from learned judgment. A text-only planner, before any frame is read, compiles verbatim prompt spans into typed physical claims and a statically validated execution plan. The surface language is small: existence, count, temporal order, spatial relation, trajectory, sound, on-screen text, plus a fallback semantic judge. The compiler fills in the dependency closure. "A before B" automatically adds occurrence gates and two timing actions.
Execution is topological. Observations may only activate, skip, or narrow calls already in the plan; they cannot invent new checks. Specialists are frozen: SAM 3 for segmentation and tracking, TAPNext++ for eleven track-based physical measurements, plus monocular depth, OCR, and audio-event detection. Each action emits a provenance-carrying record whose payload is a typed measurement or an explicitly tagged learned state. Unmeasurable cases abstain; infrastructure failure is a separate outcome, so "could not measure" is never written as "absent in the video."
Decisions are three-valued. Sub-checks roll up to claims, claims to the clip: one contradiction makes the clip implausible, all-supported is plausible, otherwise abstain. Planner and dense semantic verifier share Qwen3-VL-30B-A3B; passed claims are re-read by a closed fallback, gpt-5.6-sol. Generation is a separate map: MuJoCo simulates and validates motion, renders it as a depth control, and conditions a frozen Wan 2.2-VACE. The solver owns the motion; the prompt owns appearance. This report scores generator and critic apart. Closed-loop rewrite is only a first trial.
The human corpus has 1,500 generated clips and 2,582 flaw records. The evaluation core is 149 clips with 304 flaws. It is a development set, not a generalization test.
| Evaluator | Flaws found | Recall |
| Question decomposition (same backbone) | 164 / 304 | 53.9% |
| Single-pass VLM over the full clip | 222 / 304 | 73.0% |
| VeriPhy | 228 / 304 (191 whole + 37 partial) | 75.0% |
Of the 228 matches, 83 are cases where the fallback flipped a dense-verifier pass to contradicted. By category, counting is worst at 49%, because a per-frame series is reduced to its mode and an intermittently grounded subject reads as absent. On-screen text is 84%: the recognition head copies glyphs and will not repair a garbled banner into the prompt's wording. Physical specialists were called 227 times, abstaining on 166 and contradicting on 61. CLIPScore is uncorrelated with severity and flaw count.
A held-out 502-flaw slice of the same corpus tests in-context lessons: 33 planner lessons distilled from misses, no weight update. With lessons the critic finds 375 / 502 (74.7%) against 340 untaught (67.7%). Lessons from 25 experience clips already saturate the gain; more experience reshuffles which flaws are found. Closed-loop rewrite on unseen Cosmos3-Nano rewrites the prompt for the 87 / 589 clips the critic flags: pplausible rises on 72.4%, 49 of 65 implausible verdicts flip to plausible, and 1.6% of unflagged clips regress. VideoPhy-2's independent AutoRater on the same pairs is flat on both semantic and physical axes.
Physion-Eval numbers are interim on the first 601 clips. The evidence-conditioned physical-implausibility flag rate is 26.8%, against 52.7% for untrained humans and 16.9% for Gemini 3.0 Pro. Samples and protocols differ, so the comparison is descriptive.
Raw recall is almost tied with asking the same backbone once (75.0% vs 73.0%). What VeriPhy adds is a measurement, a time window, and provenance on every verdict, so a human can audit one decision at a time and, in principle, write the packet back into generation. For people evaluating video world models, that chain is more useful than another scalar.
The closed loop does not yet hold. The gain lives only on the critic's own score; the independent rater does not move. The paper leaves both readings open: the rewrite improved a physical quality the AutoRater cannot see, or it only raised the critic's score. The experiment cannot tell them apart.
The authors are direct. Control comes from a simulator, so scene diversity is simulator-bounded. The corpus contains only flawed clips, so precision is unmeasurable. Annotation is single-rater, with no agreement study. The core is a development set, and the question-decomposition baseline was specified after inspecting critic outputs. Binding is a central failure: the compiler can keep symbolic names consistent and still not guarantee that two tools localized the same physical instance. The count path can collapse "grounding failed on most frames" into zero. The planner has no "on top of" predicate; negated measurements fall back to the generalist verifier.
The lesson study dropped the acceptance gate that was supposed to keep a bad lesson out. The taught planner reads 1.76 times as many claims from a prompt as the untaught one, so some of the recall gain is "ask more." gpt-5.6-sol is closed; 83 of 228 matches went through it, and an open reproduction will sit lower.