VideoMemBench shows LLMs forget video chats: best model scores just 45.7 across 25 tested

机器之心 · wechat · 2026-11-21

Researchers from UTS, CUHK, HKUST and Zhejiang University released VideoMemBench, testing whether multimodal LLMs can remember and correctly use earlier context in multi-turn conversations about the same video.

Design: unlike independent QA, tasks involve two dependency types —

Built via a Planner/Generator/Judger pipeline with expert review: 326 videos, 2,476 QA turns, 1,387 requiring prior context.

Results: Claude Opus 5 leads at 45.7, followed by Gemini 3.8 Flash (42.4), Kimi K3 (42.2), GPT-5.6-sol (41.7), Qwen3.5-27B (34.3). Even the best model scores only 35.7% on answer dependency.

Findings: as dependency distance grows from 1 to 5+ turns, average accuracy drops from 30.2% to 23.1%; replacing history with correct answers markedly improves performance, showing early errors propagate.

Original post →

More from Models

Models channel →