arXiv paper reuses KV cache via M-RoPE offset for editable video context, cutting prefill tokens 30%
RulinShao · x · 2026-11-23
The paper "Editable Video Context: Mid-Context KV-Cache Reuse for Context-Managing Video LLMs" extends Context Language Models and Suffix Cache Reuse to video LLMs:
- Method: a long video is represented as timestamped segments. The model views a sparsely sampled version, selects segments to decode at 4x frame rate, and drops the rest. Surviving segments' KV caches are reused by rotating keys with a single offset shared across the temporal/height/width components of Qwen2.5-VL's M-RoPE; a content-hashed cache also skips redundant vision-encoder computation.
- Results: on Qwen2.5-VL-7B with synthetic needle-in-a-video-haystack questions over public-domain films, prefill tokens and rebuild latency drop versus full recomputation while answers usually stay the same; prefill tokens cut by 30%.
- Limitation: densifying a known answer segment improves accuracy, but when the model picks segments itself, accuracy doesn't improve — it rarely finds needles missing from the sparse sample.
Code, raw logs and reproduction scripts are released. CLM author Rulin Shao highlighted the work, suggesting RL might teach video models where to look and that native video cache reuse in vLLM/SGLang could convert the savings into real speedups.
Related event: KV-Cache Reuse Brings Mid-Context Editing to Video LLMs(2 posts)→
More from Infra
- Apple cuts per-dev daily token budget from $300 to $100; cheaper-model strategies follow — Zachly · 2026-11-23
- Custom sm_60 kernels push Tesla P100 decode from 11.6 to ~52 tok/s, repo open-sourced — Relative-Ingenuity15 · 2026-11-23
- Open-Source Maestro Video Generator Lures Users Away From $40/Month AI Subscriptions — cocktailpeanut · 2026-11-23
- A 9-machine local AI homelab: 5 agents on one dual-3090 llama-server — StandardLovers · 2026-11-23
- Prune, train, upcycle: one model that scales with your RAM — minhnguyen · 2026-11-23
- Local LLM inference is hopelessly fragmented: one user now juggles 5 engines per model — Fancy-Snow7 · 2026-11-23