HarnessEval-W: Agentifying the Evaluation of Visual Worlds
Weiliang Chen, Haowen Sun, Jun Gao, Jiawei Chi, Hanyang Wang, Qiyu Dai, Yihao Li, Hao Li, Jingnan Gao, Yi-Hsin Hung, Xingzhuo Guo, Shangchen Miao, Zhiyuan Shi, Xiang Li, Fengrui Tian, Weihua Du, Ziqi Huang, Shenyuan Gao, Siqiao Huang, Mingyu Liu, Yifei Li, Shizun Wang, Xi Wang, Tianqi Zhang, Xue Luo, Xiyin Ren, Jinshan Ren, Xiaoyang Shen, Xiaobo Hu, Zhiyang Dou, Mingyu Ding, Yichao Yan, Xinchao Wang, Yizhou Wang, Shilong Liu, Wenzhao Zheng, Yueqi Duan, Yuan Gong, Ziwei Liu, Ming-Yu Liu, Jialong Wu, Jiangran Lyu, Fangfu Liu
cs.CV
2026-08-18
HarnessEval-W turns world model evaluation into an agentic pipeline: case-specific skill routing, sub-question decomposition, and tool-grounded sub-agents produce an evidence tree behind every score. Across 18 models and 330 cases it aligns with human A/B preferences at Spearman 0.87-0.93, beating the closest WBench protocol by 10-40 points of pairwise accuracy.
Scores on world model leaderboards explain nothing. VBench-style metrics and one-shot LLM judges compress a rollout into a scalar: whether the robot gripped the right spoon, whether an extra hand appeared, whether physics held up is invisible. There is no reasoning chain to audit, no way to locate the failure, and no way to verify the number was earned.
The authors read this as a paradigm gap. A human evaluating a generated world tracks objects, checks permanence, and verifies causal constraints as a workflow. LLM evaluation long ago engineered such workflows into harnesses; visual world model benchmarking never made the jump.
HarnessEval-W decomposes evaluation into three agent layers:
Evaluation axes derive from a formal factorization of world models: Observation Quality (rendering), Transition Correctness (action execution), World Persistence (long-horizon coherence), split into eight settings. Cases are built by an agentic pipeline: taxonomy sampling, image generation, image-grounded action planning, and a separate validator that rejects and resamples weak candidates, yielding 330 cases. All sub-agents share one VLM backend, temperature, and frame sampling.
Eighteen models across three conditioning interfaces were evaluated on identical cases:
| Model | Interface | Overall |
| Seedance 2.0 | Text prompt | 75.5 |
| Wan 2.7 | Text prompt | 75.0 |
| Kling 3.0 | Text prompt | 74.4 |
| SANA-WM | Native action | 68.7 (best Offscreen Evolution, 72.3) |
| HY-WorldPlay 1.5 | Camera pose | 67.1 (best Revisit Consistency, 81.9) |
The evaluator's own validation matters more:
| Comparison | Metric | HarnessEval-W | WBench protocol |
| Physical & causal | Pairwise accuracy | 71.7% | 31.9% |
| Physical & causal | Draw rate | 1.8% | 52.2% |
| Intentional & event edit | Pairwise accuracy | 77.8% | 60.2% |
Against Bradley-Terry strengths fitted from 5,000 human A/B judgments, rank correlation reaches Spearman 0.93 (intentional) and 0.87 (physical). Re-running at temperature 0 three times, the fitted-curve envelope is 4.9x narrower than WBench's. Same GPT-5.5 backend, temperature, and frame sampling throughout; only the protocol differs.
Two side findings deserve mention. Intentional and physical transition correlate at r = 0.98, while exploratory transition is nearly unrelated to both (-0.15 / -0.18): coherent continuation says nothing about instruction following. And fine-tuning a text-to-video generator into an action-conditioned model redistributes capability rather than lifting it: Wan 2.2 to DreamX-World gains +4.8 exploratory and +7.8 revisit while losing 11.9 intentional; HunyuanVideo 1.5 to HY-WorldPlay 1.5 trades +8.4 revisit for -24.2 intentional.
For world model developers, this is an evaluation that localizes failure modes: a low score traces to a concrete sub-question instead of an opaque drop. For the evaluation community, it is the first full transplant of the harness paradigm into visual generation, open-sourced as a living benchmark with an extensible skill library.
The fine-tuning numbers carry direct selection guidance: if your application demands precise instruction execution (robotic simulation, interactive editing), general-purpose video generators still beat dedicated interactive world models, which pay for exploration and persistence out of their instruction-following budget.
The evaluator's ceiling is its VLM backend. Every sub-agent uses the same VLM, so fine-grained deformations or fast motion that the VLM itself misreads will corrupt the evidence tree no matter how clean the reasoning. The 18-model comparison runs each model's released checkpoint through its native conditioning interface; interface differences (text versus camera trajectories versus action sequences) are not separately controlled. With 330 cases across eight settings, each probe family holds only dozens of cases, and no confidence intervals are reported per metric. The human-alignment study covers nine representative models on two transition settings only. The compute cost of agentic evaluation (VLM calls per rollout) is never stated, which determines whether third parties can rerun it at all.