Same model, same benchmark, up to 19.5pp apart: Meta-Video exposes video eval chaos
HildeKuehne · x · 2026-10-15
Researchers from Tübingen, MIT-IBM and IBM Research introduce Meta-Video, showing that unexamined evaluation choices in video understanding benchmarks are a major source of variance: just the eval configuration (frame sampling, preprocessing, input formatting — 9 parameters total) can shift accuracy by up to 19.5pp for the identical model checkpoint. Reviewing 20+ papers, they found nearly 20% score gaps for the same backbone on the same dataset, meaning reported gains may reflect eval setup rather than real capability gains.
Contributions:
- Meta-Video benchmark: 1,000 video-dependent questions over 791 videos from five benchmarks, preserving model rankings at 5× less compute;
- Evaluation cards: a one-table record of every setting behind a score (backend, batch size, frames, sampling, decoder, resolution, prompt, temperature), shipped with every table in the paper;
- Open-source paper, code, and dataset.
Related event: Meta-Video Exposes Up to 20-Point Score Swings in Video LLM Benchmarks(4 posts)→
More from Models
- Step 5 Preview architecture blog details Sparse GQA with block-wise token merging cutting indexer cost 8x — teortaxesTex · 2026-10-15
- Fish Audio Drama 3 lands in vLLM-Omni: 128 concurrent streams on one H200 — vllm_project · 2026-10-15
- Why is GPT-6 Astra ranked so low on LM Arena? Outscored by GPT-5.6 and 6.1 — AreaExact7824 · 2026-10-15
- Stance test heat map shows Sonnet 5.5 resists framing manipulation other models succumb to — teortaxesTex · 2026-10-15
- Argon (Medium) burned 85% of a 5-hour usage limit in just 5 minutes — Avalunne · 2026-10-15
- Score vs cost chart puts Gemini 4 Argon on the Pareto frontier, Kimi K3 far off it — Hesamation · 2026-10-15