WOVEN Paper: World Modeling Is a Measurable Skill; 7h of Synthetic Video Beats 125k Hours of Real Data

机器之心 · wechat · 2026-11-06

A team from Northwestern, CMU, UNC and Amazon presents WOVEN, narrowing 'world modeling' to state transition p(s'|s,a) and unifying spatial, embodied, physics and temporal tasks as different queries over the same tuple. They use a video generation model as a transition engine to build a 36K-question benchmark where scene, action and inference operation can be independently varied, with error choices labeled by error type.

Measurement: Across 38 frontier models, the best (GPT-5.4) scores 65.8% vs 92.3% for humans. Failure patterns are consistent across families: temporal adjacency sits at 25–28% (near random), geometric perturbations (position/orientation swaps) drop scores to 30% while appearance perturbations barely matter, and models acquire physical principles in the same order as infants—collisions never pass. Scaling alone doesn't fix geometric robustity (+9 points from 3B to 72B on Qwen2.5-VL).

Trainable and efficient: SFT+RL on WOVEN lifts Qwen2.5-VL-3B from 26.4% to 89.3% in-domain; 7 hours of generated video for post-training beats 125k hours of real-video mid-training on 3DSRBench (+6.7 vs +4.0). Learned ability transfers zero-shot to up to 22 external benchmarks, up to +27.3 points.

A shared underlying capability, not task stacking: transfer curves of controlled subsets are pairwise correlated (r=0.86 between world-modeling gains and external gains); a counterfactual 'virtual world' where transition rules are deliberately wrong shows zero downstream gains, proving causal dependence; replacing 30–50% of a downstream task's own training data with WOVEN preserves accuracy.

Training recipe: what determines transfer is the reasoning operation over the transition, not scene/action similarity; prefer high visual-state-change supervision for robustness; temporal and agent-driven tasks need direct supervision; no transfer between active and passive actions. The recipe passed prospective validation on 7 unseen benchmarks across 8 seeds.

Original post →

More from AGI Musings

AGI Musings channel →