WOVEN benchmark finds all 38 frontier MLLMs fail visual transition reasoning far below humans
MLL-Lab · hf · 2026-10-14
MLL-Lab (Northwestern/CMU/UNC/Amazon) released WOVEN, a training source and benchmark that treats visual transition reasoning as a shared primitive for visual world modeling in MLLMs.
- Benchmark: 36,076 examples across 20 scene types, 5 action types, and 8 reasoning types, built from realistic rollouts of video-pretrained generative models.
- Findings: Evaluating 38 frontier MLLMs (GPT-5.4, Qwen3-VL-235B-A22B, etc.) shows all fall far below humans, with failures recurring across model families and scales.
- Transfer: Training subsets of 2,000 items each improve 22 of 26 external benchmarks by up to 27.3 points (spatial +27.3%, embodied +12.5%, physics +10.6%, temporal +11.7%); WOVEN data can replace 30-50% of a task's own training data at comparable accuracy.
- Recipe: Select supervision by the reasoning operation it teaches rather than the domain shown, and prefer larger visual state changes for robustness.
More from Multimodal
- Gemini team demos 24-word prompt building a synthesizer that turns horse gallops into music — GeminiApp · 2026-10-14
- Noiz AI open-sources HelixWorld, a real-time interactive audio-visual world model — SimplyAnnisa · 2026-10-14
- 'Oh no!': AI video of a man chased by a giant monster goes viral — phusa29 · 2026-10-14
- ComfyUI-EdgeTTS v2.0.0 fixes Microsoft's 403 breakage, adds native SRT export — Narrow-Particular202 · 2026-10-14
- AI-generated portrait of Charlemagne impresses with detail — MeroveeFrancSalien · 2026-10-14
- Kroma: a free ComfyUI workflow chaining Chroma and Krea 2 with 2x upscale — gabrielxdesign · 2026-10-14