Review of 20+ papers: eval settings alone swing video benchmark scores by nearly 20%
HildeKuehne · x · 2026-10-15
Hilde Kuehne's team reviewed 20+ video understanding papers and found nearly 20% score swings for the same backbone on the same datasets, caused purely by evaluation parameters.
- Varying inference backend, batch size, frame sampling, number of frames, video decoder, resolution, and decoding temperature changes results substantially
- Frame count (4→128) and input format (.jpg vs .mp4) have outsized effects on Meta-V aggregate scores; decoding temperature from 0 to 1 degrades multiple benchmarks
- Conclusion: much reported "progress" in video understanding may reflect eval configuration differences; papers should disclose full eval settings
Related event: Meta-Video Exposes Up to 20-Point Score Swings in Video LLM Benchmarks(4 posts)→
More from Models
- Why ChatGPT beats the better Claude with mainstream users: being first known — aitrendz_xyz · 2026-10-15
- 'You need high pain tolerance to review Gemini 4,' researcher quips amid error screens — gleech · 2026-10-15
- MiniCPM-V 4.7 runs a 35B MoE multimodal model on phones with just 3GB memory — nevrekaraishwa2 · 2026-10-15
- Gemini 4 Argon wrote 5,000+ lines of pure Python to build its own ray-tracing 3D engine — aitrendz_xyz · 2026-10-15
- JetBrains' thinking MoE Mellum 2.1 (12B-A2.5B) GGUF trends on Hugging Face — JetBrains · 2026-10-15
- Microsoft Accidentally Leaks OpenAI Secret: GPT-6.1 Sol Is GPT-6 Sol With Fewer Thinking Loops — evilsocket · 2026-10-15