PlayWorld: Benchmarking world models via long-horizon agent gameplay
机器之心 · wechat · 2026-08-23
HKU, Kwai, and collaborators introduce PlayWorld, a new benchmark that evaluates world models like a user "playing a game".
Core Pain Point
- Traditional fixed-trajectory benchmarks fail to ensure consistent viewpoints across models and ignore the completion of long-horizon goals.
Innovation: AgentPlayer
- Instead of fixed keystrokes, it defines a "scene" and a "long-horizon goal".
- AgentPlayer observes generated video frames in real-time and makes 5 decisions: Keep, Stop, Extend, Correct, End, adapting to different model control granularities.
Four Evaluation Dimensions
- Geometry Consistency: Do object identity, position, and texture persist after movement/revisits?
- Interaction Fidelity: Do actions follow physical laws (e.g., water ripples)?
- Out-of-sight Evolution: Does the world continue causal evolution when the target is off-screen?
- In-sight Evolution: Does the process actually advance over long observation periods?
Findings
- Bottleneck: Almost all models score poorly on continuous state evolution (in/out of sight), failing to maintain cross-temporal semantic states.
- Spatial Consistency: Long-horizon revisits reveal global spatial inconsistencies (e.g., landmarks change after 360° rotation).
- Arrival ≠ Understanding: High trajectory pass rates do not equate to high world understanding (e.g., SANA-WM has high arrival rate but low overall score).
More from Research
- VideoCoCo Fixes Video Physics by Generating Executable Code — TinfoilTricorn · 2026-08-23
- Neuroscientist: text-only training uses only half of our visual capacity — 3scorciav · 2026-08-23
- JFPuget Team Claims 100% Score on Public ARC-AGI Benchmark — JFPuget · 2026-08-23
- US National Labs Developed Monte Carlo and Foundations for Numpy — MikePFrank · 2026-08-23
- Paper proposes method for Jensen-Shannon centroid of categorical distributions — FrnkNlsn · 2026-08-23
- Building a text watermark remover for Claude, rewriting works best — Imaginary_Dinner2710 · 2026-08-23