Meta-Video Exposes Up to 20-Point Score Swings in Video LLM Benchmarks

Researchers from Tuebingen, MPI and MIT released Meta-Video, showing that evaluation settings alone—frame sampling, preprocessing, inference backend, etc.—can shift video LLM benchmark scores by nearly 20 points, after auditing over 20 papers.

2026-10-14 ~ 2026-10-15 · 4 related posts

• Meta-Video shows frame sampling, backend and temperature skew video understanding benchmarks — HildeKuehne · 2026-10-14
• Same model, same benchmark, up to 19.5pp apart: Meta-Video exposes video eval chaos — HildeKuehne · 2026-10-15
• Review of 20+ papers: eval settings alone swing video benchmark scores by nearly 20% — HildeKuehne · 2026-10-15

1 near-duplicate retellings: HildeKuehne