Meta-Video shows frame sampling, backend and temperature skew video understanding benchmarks
HildeKuehne · x · 2026-10-14
The Meta-Video project questions how video understanding benchmarks are evaluated: most papers treat frameworks like lmms-eval as black boxes, inheriting defaults for frame sampling, preprocessing, prompt formatting, and inference backends.
Per the accompanying diagram, scores are affected by:
- Inference & decoding: backend (HF transformers vs vLLM) and batch size change numerics and padding
- Frame extraction: number of frames (8-128), uniform vs fixed-fps sampling, video decoder (decord/torchvision/torchcodec), and frame resolution (200704-602112 max pixels) determine what temporal/spatial evidence the model sees
- Model input: mp4 vs jpg, prompt format, and decoding temperature (0-1.0) affect answer stochasticity
Takeaway: leaderboard gaps may reflect eval configuration rather than model capability.
More from Multimodal
- RVM: Likelihood-Free Reward-Based Velocity Matching for Scaling Diffusion RL — gowthami_s · 2026-10-14
- Everygen Wizard Launches: Full Video Workflow in One Conversation — Scobleizer · 2026-10-14
- Tiny Recursive Reasoner Boosts Diffusion Models: Sudoku 2.9% to 71.2% — jm_alexia · 2026-10-14
- ElevenCreative Recast Swaps Actors in Footage With One Reference Photo — SucceededMind · 2026-10-14
- Explaining how diffusion models work, using beavers — manicman1999 · 2026-10-14
- IsoVRBench: 160K synthetic video pairs via targeted interventions train reward models without human labels — StellaLisy · 2026-10-14