IsoVRBench: 160K synthetic video pairs via targeted interventions train reward models without human labels

StellaLisy · x · 2026-10-14

The paper introduces IsoVRBench: perturbing anchor videos along specific dimensions (frame shuffling, frame holding, and eight interventions total) yields fine-grained preference labels by construction—no human annotation needed. Key findings:

Existing video reward benchmarks suffer from pathological generator-identity dependence: generator identity alone predicts 74–78% of human labels, so reward models degenerate into generator identifiers and fail to generalize.
SOTA reward models and VLM judges fail on specific IsoVRBench dimensions (notably TemporalShuffle/FrameHold—models rely on frame quality and ignore frame ordering) while humans perceive these differences nearly perfectly.
IsoVRM trained on 160K synthetic video pairs matches a model trained on 180K human labels, with additional gains when synthetic data stacks on top of human data.
Post-training Wan2.2-TI2V-5B with the reward model improves VBench-2.0 scores and human preference.
Feeding more frames into context does not fix temporal consistency, suggesting architectural changes are needed.

Original post →

IsoVRBench: 160K synthetic video pairs via targeted interventions train reward models without human labels

More from Multimodal