FIRM-Video: Check Before You Score for Reliable Text-to-Video Reward Modeling
Peiyuan Zhang, Xiangyu Zhao, Hongbo Liu, Xiaoxing Hu, Mingxin Liu, Shuran Ma, Yunhang Shen, Jian Hu, Haihan Gao, Haoyu Cao, Xue Yang
cs.CV
2026-08-22
FIRM-Video verifies checklist items before scoring, then distills an 8B reward model. Overall MAE 0.78 vs 1.33 base; Best-of-8 leads VBench totals on three generators.
A text-to-video reward model has three jobs: did the clip follow the prompt, does the world hold together, and does it look clean. Most current judges score holistically, either with a fixed rubric or with free-form chain-of-thought. Three failure modes follow. Inspection is incomplete: brief actions, object relations, and local defects slide by. Justifications are unfaithful: the score is chosen first and the prose is backfilled. Attribution is entangled: one glitch is billed once as a semantic miss, once as physics, once as a visual defect. Multi-stage pipelines are more careful and too expensive to serve. End-to-end models are cheap and starved of verifiable labels.
FIRM-Video turns check-before-score into a data factory. Each dimension has its own checklist source.
Instruction Following: Qwen3-32B splits the prompt into atomic yes/no questions with importance 1 to 5, shared across videos of that prompt. A multimodal verifier answers against temporally ordered frames; unverifiable items count as misses. The score is an importance-weighted satisfaction rate. World Coherence: Qwen3-VL-235B, without the prompt, inventories entities and actions and assigns importance, then a planner writes structure, motion, and physics questions per target. The prompt is used only to tell intended fantasy or style from real violations. Scores aggregate a per-target clean ratio, so content-rich clips are not punished for attracting more questions. Perceptual Quality: a human-curated defect taxonomy; the prompt only separates stylized lighting from rendering errors.
Structured decisions are then verbalized. The numeric score is locked before the rationale is written. Running this expensive pipeline offline yields FIRM-Video-90K: 29,348 videos, 3,012 prompts, 88,044 dimension-specific items, drawn from Rapidata and VideoFeedback2 across 20-plus generators. Qwen3-VL-8B and InternVL3-8B are fully fine-tuned for two epochs on eight uniformly sampled frames. At inference the student does one forward pass, with no planner, grounder, or verifier.
FIRM-Video-Bench holds 250 videos and 750 expert point scores, no overlap with training, and no rating level above 30.4% in any dimension.
Qwen3-VL-8B's overall MAE drops from 1.33 to 0.78 after training, with strict/relaxed accuracy 0.42 / 0.84, beating GPT-5 (1.08), Gemini-3.1-Pro (1.16), and Doubao-Seed-2.0-Lite (1.05) on the overall number. IF MAE is 0.65, still behind GPT-5's 0.62; WC MAE 0.86 is the lowest in the table. The InternVL3-8B student lands at 0.85 overall. The offline teacher pipeline itself is 0.73, so distillation is lossy.
On MJ-Bench-Video preference accuracy, the InternVL3 student hits 31.93 Alignment and 35.35 C&C, 31.06 overall, near Gemini-3.1-Pro's 33.06. In Best-of-8, the Qwen student posts the best VBench Total, Quality, and Semantic scores on LaVie-Base, CogVideoX-2B, and Wan2.1-T2V-1.3B. Versus the next-best selector, Total rises 0.27 / 1.01 / 0.66 and Semantic 1.24 to 2.11. On LaVie, scaling N from 2 to 16, it leads for every N≥4; at N=16 it is 1.03 above random and 0.33 above untuned Qwen3-VL.
Reward-model quality is often a labeling-contract problem, not a student-size problem. Checklist, verify, then aggregate compresses a multi-stage judge into one 8B forward pass, usable as a Best-of-N selector and later as an RL reward. Dimension-wise scores also map onto VBench's split axes without collapsing to a single scalar. An 8B model beating larger closed models on overall MAE is the argument that the annotation contract is worth more than scale. Incremental work: expensive data, cheap students.
The authors have not yet trained a generator against this reward; Best-of-N and pointwise agreement are not an RL loop. Median clip length is 3.3 seconds, mostly 1 to 6 seconds, median prompt 27 words, so long shots and messy temporal structure are barely tested. Perceptual Quality labels skew high, which matches current generators looking mostly fine, and also flattens PQ discrimination. Data construction depends on a proprietary multimodal verifier and a 235B grounder. On MJ-Bench the Qwen student's Overall is 27.81, not a clean sweep. Eight-frame sampling can still miss a short action.