Meta-Video: Eval config alone shifts video model scores by up to 19.5pp, paper finds
HildeKuehne · x · 2026-10-15
Researchers from Tübingen, MPI, and MIT CSAIL released Meta-Video (arXiv:2610.18535), a study plus benchmark showing video-LM benchmark scores are far less comparable than assumed, with paper, code, and data all public.
Key findings
- Benchmarks are usually run through frameworks like lmms-eval with inherited defaults. Unexamined choices — inference backend (HF vs vLLM, ±3%), input format (mp4 vs jpg, ±4%), video decoder (±5%), frame sampling (±3%), decoding temperature (±5%) — each shift results; the same checkpoint can vary by up to 19.5pp from eval configuration alone, undermining cross-paper comparisons.
- Many benchmark samples are answerable without watching the video, and full suites are expensive to run.
Contributions
- Meta-Video: a balanced 1,000-question benchmark over 791 videos curated from Video-MME, LongVideoBench, CVBench, VideoMMMU, and VSI-Bench. It largely preserves model rankings and gives a 5× inference speedup for full frame-count sweeps (4–128 frames); the authors claim reliable performance trends in 20% of the usual benchmarking time, with clearer trends than current averages.
- Evaluation Cards: a proposed best practice — standardized tables capturing backend, frame settings, decoder, resolution, temperature, and prompts — to be linked in paper supplements for reproducibility.
Related event: Meta-Video Exposes Up to 20-Point Score Swings in Video LLM Benchmarks(4 posts)→
More from Models
- Why ChatGPT beats the better Claude with mainstream users: being first known — aitrendz_xyz · 2026-10-15
- 'You need high pain tolerance to review Gemini 4,' researcher quips amid error screens — gleech · 2026-10-15
- MiniCPM-V 4.7 runs a 35B MoE multimodal model on phones with just 3GB memory — nevrekaraishwa2 · 2026-10-15
- Gemini 4 Argon wrote 5,000+ lines of pure Python to build its own ray-tracing 3D engine — aitrendz_xyz · 2026-10-15
- JetBrains' thinking MoE Mellum 2.1 (12B-A2.5B) GGUF trends on Hugging Face — JetBrains · 2026-10-15
- Microsoft Accidentally Leaks OpenAI Secret: GPT-6.1 Sol Is GPT-6 Sol With Fewer Thinking Loops — evilsocket · 2026-10-15