RoboJudge: 800 human-annotated videos benchmark MLLM judges for embodied video generation
CUHK-CSE · hf · 2026-10-26
Researchers from CUHK, UCSD, Northeastern, and Zhejiang University introduce RoboJudge-Bench, a benchmark evaluating multimodal LLMs as judges of embodied video generation.
- Video generation models are promising foundations for embodied world models, but their rollouts must be physically valid and faithful to task instructions—requiring reliable automated evaluation.
- The benchmark measures Physical Adherence and Instruction Alignment across 800 human-annotated videos from 8 source corpora and 10 video generators, spanning grippers, human hands, and dexterous hands.
- Experiments show uneven judge-human alignment: general-purpose MLLMs score lower on physical adherence, while specialized video judges perform better.
- The team built a training dataset from the benchmark and fine-tuned a 9B RoboJudge model with high agreement with human ratings.
More from Embodied
- Microduck team publishes deep dives: relocalizes within 2cm using a 64-pixel depth sensor — mishig25 · 2026-10-26
- Robot's elbow-assisted get-up move goes viral for its humanlike fluidity — iamfakhrealam · 2026-10-26
- Chinese eVTOL EHang EH216-S completes nearly 100,000 flights across 23 countries — LoganGrasby · 2026-10-26
- Toyota bets $6.4 billion a year on humanoid robots and factory automation — The Decoder · 2026-10-26
- Robot runs a video generation model as its policy: frozen FLUX beats VLAs — lukas_m_ziegler · 2026-10-26
- XPeng clears Guangzhou L4 robotaxi trials, gates rides behind invite codes — VectorWireAI · 2026-10-26