PAWBench Results: Even Told the Target Outcome, Video Models Hit It Only 38–58% of the Time
RisingSayak · x · 2026-08-28
Key results from the PAWBench thread: the authors tried three fixes—clearer outcome instructions, more diverse sampling, and additional training—each helping a different part of the problem, none solving it end to end. Even when given the desired outcome, models produced it only 38–58% of the time. Models also react to the wrong cues: changing actual physics moves predictions too little, while irrelevant visual or text cues move them too much.
More from Research
- SubQ open-sources Decision LCR, a 72K–115K-token benchmark for decision models — ryanallred · 2026-10-16
- Interpretability debate: 'features' as fundamental neural network units may be wrong — MoonL88537 · 2026-10-16
- FOMO-Dex adds force-aware compliance to dexterous teleoperation for contact-rich robot tasks — HaozhiQ · 2026-10-16
- New mech interp hypothesis: neural networks generalize from the very first training step — burny_tech · 2026-10-16
- Princeton launches research initiative on keeping autonomous AI systems aligned with human intent — LakeBrenden · 2026-10-16
- OrganicHAR at UbiComp 2026: Discovering Activities Sensors Can Actually Distinguish — mynkgoel · 2026-10-16