Character.ai says pairwise judges catch video drift better than absolute scores

AI Engineer · youtube · 2026-07-25

Maor Bril from Character.ai argues that video evaluation cannot rely on single absolute scores: a clip can look polished while still failing on temporal coherence, shot continuity, or story consistency.
Traditional metrics like CLIP miss these failure modes, and human review does not scale.
The team found pairwise preference judgments more robust than absolute scoring, and trained a Qwen3-VL judge with Bradley–Terry loss on pairs of real and deliberately broken footage.
The judge is used as a CI regression gate for every AgentX release at Character.ai, calibrated against human scores, so drift is caught early before users see it.

More from Apps