VideoMemBench shows LLMs forget video chats: best model scores just 45.7 across 25 tested
机器之心 · wechat · 2026-11-21
Researchers from UTS, CUHK, HKUST and Zhejiang University released VideoMemBench, testing whether multimodal LLMs can remember and correctly use earlier context in multi-turn conversations about the same video.
Design: unlike independent QA, tasks involve two dependency types —
- Episodic dependency: questions refer back to a previously discussed scene;
- Answer dependency: follow-ups on entities the model identified earlier, scored chain-wise (wrong earlier answers void later ones).
Built via a Planner/Generator/Judger pipeline with expert review: 326 videos, 2,476 QA turns, 1,387 requiring prior context.
Results: Claude Opus 5 leads at 45.7, followed by Gemini 3.8 Flash (42.4), Kimi K3 (42.2), GPT-5.6-sol (41.7), Qwen3.5-27B (34.3). Even the best model scores only 35.7% on answer dependency.
Findings: as dependency distance grows from 1 to 5+ turns, average accuracy drops from 30.2% to 23.1%; replacing history with correct answers markedly improves performance, showing early errors propagate.
More from Models
- Mathematician Levent Alpöge claims proof of Riemann Hypothesis co-discovered with Claude — burny_tech · 2026-11-21
- Google reportedly changing privacy and AI training opt-in controls — blackice193 · 2026-11-21
- Tao announces AI-assisted proofs: Euler/Navier-Stokes blowup and Generalized Riemann Hypothesis — burny_tech · 2026-11-21
- 7.5M TTS model holds prosody on long sentences: kale-tts-nano f0 std 2.97 st matches 45.7M sibling — NERDDISCO · 2026-11-21
- OVH's Olala Mini 7A1B: 360 tok/s decode on one RTX 5090, 96K context, v2.0 to exceed 100B — JiliJeanlouis · 2026-11-21
- Inkling 2 Mini on a 4090: catches citations near-Haiku level, 2-3x slower than Qwen — strickvl · 2026-11-21