PhyRubric: Rubric-Grounded Diagnostic Evaluation of Physical Commonsense in Text-to-Video Generation
Huiyi Chen, Weitai Kang, Shicheng Fan, Xinshuo Wang, Zhixiang Shen, Yan Yan, Philip S. Yu, Baoxin Li
cs.CV, cs.AI
2026-11-10
PhyRubric scores text-to-video clips from per-caption checklists. Seedance 2.5 is both faithful and physical on only 44.2% of clips; the best open model hits 14.9%.
Text-to-video models are pitched as world simulators, so a clip has to obey everyday physics: objects should not appear or vanish on their own, and a lamp should light when its switch is turned. Existing benchmarks ask annotators for a single score from 1 to 5. Nothing maps a violation onto a number, so the same clip can land several points apart, and the score never names the broken principle.
VideoPhy and VideoPhy-2 stay mostly inside mechanics. Heat and electricity are thin, acoustics is almost absent, and some prompts require textbook laws such as energy conservation.
PhyRubric writes a rubric for each caption before any video exists, then reuses it for every model. Scores stay comparable, and one odd clip cannot rewrite the criteria.
Semantic adherence (SA) and physical commonsense (PC), both from 1 to 5, come from VideoPhy-2, but they are computed from item verdicts. SA items are the visible facts the caption demands: main objects and events are primary; qualifiers, initial state, and order are secondary. PC items state which visible signs count as violations. Severity follows the order in which infants acquire physical knowledge. Fatal means the scene is impossible as a whole. Core means the caption's central cause and effect fails or runs backward. Major means a clear error outside that interaction. Minor means a color or texture drift that leaves the event intact.
Each PC item has an Expect line and a Fail line: where to look, and which signs count as failure. SA is pass or fail. PC can also be not shown, when the named moment never appears. An unturned switch is not scored as "turned, yet the lamp stayed dark"; the missing action is already charged to SA. A failure has to match a listed sign and carry a timestamp.
The map to 1–5 is fixed. For SA, unrelated is 1, all items passing is 5, only secondary failures is 4, exactly one primary failure is 3, and two or more is 2. For PC, no failures is 5. Otherwise the worst failure sets a cap of 1, 2, 3, or 4 for fatal, core, major, and minor, and each extra failure at major or above subtracts one. Items judge direction and order, not speed or magnitude, and do not penalize a heat source or power supply left off screen. Every severity appears at least once, so scores 1 through 5 are all reachable.
PhyRubric-Bench spans five domains, 34 categories, and 245 captions. Each caption states the visible outcome of one everyday causal event. Clips are silent, so acoustics covers only visible consequences such as ripples and vibration. Claude Fable 5.1 and GPT-5.6 Sol drafted the captions; rubrics were program-checked and then reviewed by people. Ten generators produced 2,433 clips of five to six seconds. Safety filters rejected 17 more. An agent pre-labeled every item, and humans checked the verdicts.
PhyRubric-Train uses 947 captions with no shared scenes, four generators, 3,788 clips, and 45.9K item verdicts. The annotator, Claude Sonnet 5.5, zooms to full frame rate on suspicious moments so a one- or two-frame glitch is not lost to uniform sampling. PhyRubric-Judge is a LoRA fine-tune of Qwen3-VL-8B with the vision encoder frozen. At inference it sees the caption and 16 frames, not the rubric.
The headline number is joint performance (JP): the share of clips with both SA and PC at least 4. PC is averaged only where the event actually appears.
| Model | Mean SA | Mean PC | JP |
| Seedance 2.5 | 3.77 | 3.99 | 44.20% |
| Veo 3.1 | 3.68 | 3.57 | 36.20% |
| HunyuanVideo-1.5-8B | 2.70 | 3.46 | 14.93% |
| Wan2.2-T2V-A14B | 2.84 | 3.34 | 14.87% |
| LTX-Video-2B | 2.27 | 3.04 | 4.07% |
Seedance 2.5 reaches 44.20% JP and Veo 3.1 reaches 36.20%, about two to three times HunyuanVideo-1.5-8B at 14.93%. More than half of the proprietary clips still fail to be both faithful and physically plausible. Every open model has mean SA below 3, and mean PC on assessable clips never exceeds 3.5. Optics is the strongest domain for eight of the ten models; heat and acoustics are the weakest. On acoustics, Seedance scores 30% JP, Veo 15%, and Wan2.2-TI2V-5B and LTX-2.3-22B score 0%.
SA equals 2 for 50.8% of clips, so at least two primary facts are missing. Only 5.0% score exactly 4. Once the event is fully shown, physics is usually fine: 74.1% of those clips also have PC at least 4. Of 2,433 clips, 412 clear both bars (16.9%), 144 show the event but break physics (5.9%), 609 never show the event (25.0%), and 1,268 show it only in part (52.1%). On assessable clips, PC is bimodal: 52.9% score 5, 40.8% score 1 or 2, and scores 3 and 4 together are 6.3%.
Open models skip the event on 20.0% to 44.5% of clips, with LTX-Video-2B highest. Seedance and Veo sit at 5.7% and 8.6%. Scoring skipped clips the old way, as if a missing event were physically fine, lifts open-model PC by 0.16 to 0.59. LTX-Video-2B would then pass Cosmos-Predict2.5-14B and Wan2.2-T2V-A14B. Cosmos-Predict2.5-2B would rank just behind Seedance and ahead of Veo on PC, while its JP of 6.69% is eighth of ten.
PhyRubric-Judge soft scores correlate with humans at 0.61 on SA and 0.32 on PC (AUC 0.84 and 0.68), matching Gemini 4 Argon at 0.60 and 0.31. Qwen3-VL-32B reaches 0.49 on SA. VideoPhy-2-AutoEval reaches 0.23 on PC after two-stage training on 62K human ratings. Zero-shot open VLMs sit at PC AUC 0.53 to 0.58. On six generators held out of training, correlation drops by 0.03 and 0.06. Holding out one domain costs at most 0.04 on SA and 0.09 on PC, the latter on mechanics.
Of the violations, L1 core object principles are 41.6%, L2 everyday dynamics 50.7%, and L3 attribute experience 7.6%. Missing consequence (24.0%) and permanence and count (20.4%) lead. A squirt of ketchup becomes a full bowl between 1.50 s and 1.62 s while the spoon jumps; a toy fire engine's roof light is already on before the finger reaches the button at 2.30 s.
Semantic adherence is the bottleneck. Half the clips miss at least two primary facts, yet 74.1% of fully shown events are physically plausible. PC scores of 3 and 4 are only 6.3% combined, so the breaks sit in object permanence and causal order, not surface detail.
PC alone rewards avoidance. Under older protocols, open-model PC rises by as much as 0.59 when clips that never attempt the event are treated as plausible. JP separates "never depicted" from "depicted and physically wrong."
The 8B judge needs no rubric at inference and matches Gemini 4 Argon on correlation. Reconstructed JP ranks agree with humans at Pearson 0.93 and Spearman 0.87, with a mean absolute error of 4.65 points. It swaps the two proprietary models and overestimates the weakest open ones. That is enough to see which failure types pile up, and not enough to close a case on a single clip.
The rubric is an analysis tool, not a settled standard. PC agreement is still modest: correlation 0.32, AUC 0.68. The main table uses soft scores. Gemini 4 Argon's integer scores look stronger because its thinking trace cannot be disabled; a paired bootstrap of 2,000 resamples puts every such gap's 95% interval across zero. Gemini passes about 10% of clips on SA and 3% jointly, against about 20% and 14% for humans, so binary accuracy is padded by the mass of failures.
Humans overturned 2.8% of SA verdicts and 5.6% of PC verdicts. About 61% of the PC edits were violations the agent had missed. Scores were unchanged on 88.1% of clips, and on the double-reviewed subset agreement was 95.3% (Cohen's kappa 0.87). Training labels come from the same agent. The paper does not report whether the judge's diagnosis text and timestamps are right.
Captions state the outcome in advance, so open-ended physical prediction is untested. Each prompt yields one clip of five to six seconds, and resolutions from 480p to 768p are mixed (Seedance at 480p, Veo at 720p). Acoustics has 20 evaluation captions and only visible consequences. PC ignores speed and magnitude.