Frozen video encoders' prediction errors track appearance, not physical surprise, study finds
Anton Hristov · hf · 2026-11-03
- The study probes frozen video representations (V-JEPA2-G, VideoMAE2-G) to test whether latent prediction error reliably measures physical surprise.
- Using ball-platform scenarios with lawful vs physics-violating pairs (solids falling through, unsupported hovering), it finds prediction errors are driven mostly by appearance and motion cues: hue, brightness and recolor perturbations evoke errors an order of magnitude larger than physical violations.
- Layer-wise probes show the "physical projection" of errors decays sharply with encoder depth, and fixed-threshold balanced accuracy sits around 52-54%, near chance.
- Conclusion: latent prediction error is a weak basis for claims that video world models grasp intuitive physics.
More from Research
- UChicago lab demos RingIO: a ring that uses touch as input and EMS output — plopesresearch · 2026-11-03
- Book share: Quantum Geometry, Matrix Theory, and Gravity by Steinacker — FrnkNlsn · 2026-11-03
- Sebastian Raschka Maps October's 'Open October': 10 Open LLM Architectures in One Month — rasbt · 2026-11-03
- TUM improves SVD-based low-rank LLM compression via layer-wise error models and Fisher stats — TUM · 2026-11-03
- New paper: AI can forecast economics RCT results; $1M+ forecasting competition launches — Afinetheorem · 2026-11-03
- Simple fix for training collapse: momentum SGD on the LM head only — micahgoldblum · 2026-11-03