PhysVista benchmark: best VLM reaches only 0.61 correlation when rating physical plausibility

PhysVista: Benchmarking Physical Intelligence in VLMs via a Perception-Reasoning-Assessment Loop

Xinge Peng, Yiting Lu, Tianwu Zhi, Wen Wen, Jianzhao Liu, Xin Li, Zhibo Chen

NeurIPS 2026 (Main Track)

cs.CV

2026-10-01

PhysVista runs 16 VLMs through a seeing-reasoning-assessment loop on real and generated videos; perception holds up but plausibility scoring collapses, with a best SRCC of 0.61.

What problem this solves

Can VLMs reason about physics, or do they stop at scene recognition? Existing benchmarks each cover one slice. PhysBench stops at perception and contains a language shortcut: models can answer without watching the video. IntPhys2 compresses everything into binary plausible/implausible judgments, which only probes shallow intuition. PAI-Bench adds reasoning but skips the final stage, assessment. Human physical understanding works as a loop: see the state, reason about the cause, judge whether the motion obeys physical law. Testing fragments of that loop separately makes it impossible to tell where a model breaks.

A second gap had barely been measured. AI-generated video is everywhere and full of subtle physical errors, and whether VLMs can referee that content decides whether physical-quality checks can be automated at all. PhysVista (USTC, ByteDance, City University of Hong Kong, NeurIPS 2026) targets both problems.

Method

The benchmark follows a seeing-reasoning-assessment loop with three task groups.

• Seeing: spatial state perception (object relationships, interactions, scene conditions, spatial feasibility), second-level localization of the interval where a physical violation occurs, 10-class camera motion recognition, quantitative scale estimation answered as fractions such as 1/3 instead of adjectives, and uncertainty awareness (which physical parameters or outcomes cannot be determined from the frames).
• Reasoning: reconstructing event order from shuffled frames, plus five causal tasks: mechanism reasoning, principle-violation identification, dynamics prediction, counterfactual reasoning (conditions changed implicitly, e.g. an icy surface swapped for a cobbled road), and violation critique.
• Assessment: rating a single video's physical plausibility from 1 to 5, and pairwise comparison of which video is more plausible.

Three design choices carry the weight. Event-level and scale-level reasoning are separated, so the benchmark can catch a model that gets the event right but cannot distinguish magnitude, quantity, or position. Language shortcuts are shut down: every task uses one fixed, content-agnostic template, LLM-styled samples get rewritten by hand, and a video-blind pass confirms questions cannot be answered from text alone. The data mixes real videos (WISA-80K) with generated ones (VideoPhy-2, which ships with 1-5 plausibility labels); camera motion labels come automatically from Multi-CamVideo pose trajectories. Some ground truth was drafted by Gemini 3.1 Pro and then human-verified.

Results

Sixteen models, from GPT-6 Sol and Claude Opus 5.5 down to Qwen3-VL-8B:

Four findings stand out. Rankings swing across stages: GPT-5.2 ranks 3rd on perception average but 14th of 16 on reasoning, and Gemini 3.1 Pro ranks 3rd on reasoning yet its score correlation of 0.3087 lands in the bottom half. Strong perception does not buy strong reasoning, and neither buys calibrated assessment. Counterfactual reasoning fails across the board: the best model reaches 49.58% and all 16 sit below 50%, while temporal ordering is comparatively easy (Doubao Seed 2.0 Pro hits 91.33%). Several models collapse onto a single score: Qwen3-VL-8B and Kimi K2.5 stamp most videos with the top rating, while Gemini 3.1 Pro and Grok 4 default to the lowest, meaning apparent accuracy on real videos may just be the default colliding with the label. Generated videos amplify the weaknesses: uncertainty awareness and dynamics prediction drop hardest on generated content, while scale estimation is slightly better there, which the paper attributes to cleaner geometric layouts.

In pairwise comparison, real-vs-generated is easiest, generated-vs-generated is clearly harder, and real-vs-real exposes a bias: when both videos are physically valid, models still force a choice instead of calling a tie.

Why it matters

If you use VLMs as judges for generated video, calibrate before trusting them. The best score correlation available is 0.61, with systematic collapse underneath, so absolute scores carry little information; relative comparison is the sturdier signal and should be the default in evaluation pipelines. On the training side, counterfactual and scale-level reasoning is the clearest gap to target with data. PhysVista invests in task formulation and shortcut-proofing rather than new data collection, and that methodology transfers to other evaluation domains.

It is a diagnostic instrument, not a model improvement. What it delivers is a fault map.

Limitations

• The main text never states the total question count or per-task sample sizes; those live in an appendix, so the statistical power behind fine-grained conclusions cannot be checked.
• Part of the ground truth was drafted by Gemini 3.1 Pro, which is also one of the 16 models under test. Human verification mitigates this, but residual bias is not quantified.
• All real-world videos are assigned the top plausibility score of 5, a coarse assumption given that effects and editing can break physics in real footage too.
• Camera motion clips are restricted to a single dominant motion, far from real mixed-cut footage.
• Model comparisons come as point estimates with no confidence intervals, so rankings separated by one or two points are not stable.

Terms

• VLM: Vision-Language Model, a multimodal model that processes images or video together with text.
• SRCC/PLCC: Spearman and Pearson correlation between model ratings and human labels, the core metric for the scoring task.
• Video-blind evaluation: testing with questions only and no video, used to detect language shortcuts in a benchmark.
• Event-level vs scale-level: event-level reasoning asks what happened; scale-level asks for fine distinctions in magnitude, quantity, or position.
• Score collapse: a model concentrating most predictions on one rating level, losing the ability to discriminate.

Source

• arXiv 2610.00559v1
• PDF
• Hugging Face
• GitHub · https://github.com/Helen1p/PhysVista

Related papers

• Same-time reallocation lifts HunyuanVideo sparse attention by 11.8 dB PSNR
• Frozen ALPS tokens hit 85.5% on LIBERO while matching V-JEPA 2-L
• Training-free segmentation: a frozen MLLM draws B-spline masks itself, 71.5 cIoU on ReasonSeg
• Wrong tap, same login screen: LGWM hits 0.987 AUC on GUI transitions in 17 ms
• Asymmetric distillation lifts 10-turn consistency from 3.03 to 3.85
• Best VLM maps scenes at 41.71% mean IoU; only 3.25% clear IoU 0.9

All paper explainers