PlayWorld: Benchmarking World Models with Agent Players over Long-Horizon Objectives
Kaixin Ding, Xi Chen, Minghong Cai, Zhiyuan Xu, Yiyang Wang, Yuxiang Lu, Junyi Li, Shuyang Chen, Yuan Gao, Xin Tao, Pengfei Wan, Hengshuang Zhao
cs.CV
2026-08-14
PlayWorld uses an Agent Player to pursue long-horizon goals in world models across 171 scenarios, finding all nine tested models unreliable at spatial consistency.
Evaluating interactive video world models (systems like Genie 3 or HappyOyster that generate video in real time in response to user actions) has a comparability problem. Humans typically judge a world model by pursuing a high-level goal through interaction — turn around 360 degrees to check the environment stays consistent, or walk into water to see if it splashes realistically. But different models respond to the same low-level commands with wildly different magnitudes: three turn-right presses might complete a full rotation in one model and only a partial turn in another. Driving every model with the same fixed action sequence therefore produces incomparable results, and closed-source models like Genie 3 and HappyOyster are only reachable through a web interface, making manual human testing expensive and slow.
PlayWorld introduces an Agent Player: a multimodal agent (built on Claude or Gemini) that simulates a human tester. Each test case ships with a human-annotated basic action sequence as a shared starting reference, so the agent isn't planning from scratch every step (which would be slow and produce inconsistent trajectories across models). At each step, the agent observes the generated frame, its own action history, and the objective, then decides to Keep, Stop, Extend, Correct, or End the current action, adapting the number and duration of actions to how each model actually responds.
Evaluation covers four dimensions: geometry consistency (does the scene structure survive a full rotation or a leave-and-return), interaction fidelity (does the subject collide realistically with walls, does water produce ripples), out-of-sight evolution (does something continue evolving plausibly while off-screen and reappear consistently), and insight evolution (does a stationary 60-second observation show natural, ongoing change). Each case comes with a tailored set of Yes/No VQA questions, scored by Gemini 3.1 Pro and aggregated into a 1-5 score. The benchmark spans 171 human-annotated cases covering 50 action patterns, producing over 1,400 interactive videos and 820+ questions across nine evaluated world models.
Among the nine models, closed-source Genie 3 scores highest overall (2.12/5); the best open-source model is LingBot-World2 (1.82). Breaking down by dimension:
| Model | Geometry | Interaction | Out-of-sight | Insight |
| Genie 3 | 2.74 | 2.40 | 1.81 | 1.51 |
| HappyOyster | 2.54 | 2.15 | 1.54 | 1.47 |
| Matrix-Game-3.0 (lowest) | 1.30 | 1.25 | 1.00 | 1.00 |
None of the nine models breaks 2.0 on insight evolution; even the best, LingBot-World2, only reaches 1.95, showing that sustaining natural ongoing change under a fixed camera is hard across the board. Trajectory validity, whether a model actually reaches the intended state, ranges from 87.1% for Genie 3 down to 41.6% for HY-WorldPlay, meaning several models simply fail to get where the objective required. 600 human pairwise comparisons correlate positively (Spearman) with the automated VQA scores across every dimension, suggesting the automated protocol tracks human judgment reasonably well.
For teams building interactive world models, this offers a reusable evaluation template: instead of testing whether a model precisely executes a fixed action sequence, test whether it can reach the same high-level objective, which puts models with different action granularities on comparable footing. The 171-scenario, four-dimension benchmark is publicly available and can be used directly against these nine baselines. The more concrete signal: even the strongest models, Genie 3 and HappyOyster, only score around 1.5 out of 5 on out-of-sight and insight evolution, the two dimensions measuring whether the world keeps behaving sensibly when you're not looking at it directly. That's a clear warning sign for anyone hoping to use world models for long, immersive applications like games or embodied-agent training environments.
The Agent Player's decisions are made by an off-the-shelf multimodal model (Claude or Gemini), and the paper doesn't test whether swapping in a weaker or stronger agent model would materially change the relative ranking of evaluated world models; there's no ablation isolating the Agent Player's own capability from the measurement. The VQA scoring also uses Gemini 3.1 Pro, and the paper doesn't discuss potential bias if a future world model happens to come from the same lab as the judge. With 171 cases split across nine models, four dimensions, and multiple scenario types, individual sub-category sample sizes are fairly small, and no confidence intervals are given for score stability.