ShallowStream: Index Shallow then Answer Deep for Streaming Video Understanding
Jitai Hao, Ke Yang, Qiang Huang, Jun Yu
cs.CV, cs.CL
2026-09-03
Shallow MLLM layers index the stream; full depth runs only on retrieved evidence. Matches top OVO-Bench methods, cutting per-frame prefill / 10s latency by up to 52.1× / 11.9×.
Streaming video is an asymmetric workload. Frames keep arriving; questions show up now and then. KV-centric systems such as ReKV, HERMES, and InfiniPot-V still run every incoming frame through the full language stack before anyone has asked anything. Later cache compression can shrink GPU memory. It cannot refund the prefill already spent, and it still materializes deep KV states that later queries may never attend.
Query-agnostic merge and eviction can drop the one moment a retrospective question needs. Stuffing the whole history into context slows the answer and can hurt current-scene perception. On-demand routers such as OASIS and WeaveTime often need an answer-level forward pass just to decide whether to look back. The unused axis is model depth.
On LVBench, Qwen3-VL-8B already retrieves well at layer 4 of 28, and LLaVA-OneVision-7B at layer 3 of 32. Shallow layers can index. Full depth can wait until a question names the evidence.
ShallowStream splits always-on stream processing from occasional answering. No task training. It rewrites the inference path of a pretrained MLLM.
During the stream, each video unit runs only through shallow layers [0, P). The shipped setup uses P=5 and 2-frame units for Qwen3-VL, P=4 and 1-frame units for LLaVA-OV, both at 1 FPS. Shallow KVs live in host memory as a full-history visual index, together with the input-level visual states so a selected unit can re-enter the language model from layer 0. No full-depth KV is stored. A short recent window is always kept as current-scene context; older units stay at unit granularity by default. Past a memory budget, optional long-cluster folds temporally adjacent, similar units into a cluster, running-averages the KVs, and keeps one representative per cluster.
A query takes three steps.
Evaluation is training-free on OVO-Bench and StreamingBench. Timing uses one RTX 5090, Qwen3-VL-8B, and five long videos.
| Backbone | Method | Real-time | Backward | OVO avg |
| Qwen3-VL-8B | ShallowStream | 80.9 | 58.1 | 69.5 |
| Qwen3-VL-8B | OASIS | 78.1 | 57.2 | 67.7 |
| Qwen3-VL-8B | SimpleStream | 80.2 | 52.5 | 66.4 |
| LLaVA-OV-7B | ShallowStream | 75.4 | 49.0 | 62.2 |
| LLaVA-OV-7B | SimpleStream | 71.0 | 49.6 | 60.3 |
On StreamingBench real-time understanding, the Qwen3-VL variant scores 78.2 against HERMES at 76.6; LLaVA-OV scores 75.5 against CausalMem at 74.3. Turning long-cluster on barely moves the totals (69.2 and 62.3).
Latency is the claim that matters. Against the heaviest baseline, per-frame prefill drops by up to 52.1× and 10-second end-to-end latency by up to 11.9×. At P=5, stream prefill is 10.72 ms/frame; P=19 costs 15.53 ms/frame and does not improve OVO backward (58.27 at P=5 is the peak of that sweep). At an 80-second query interval, ShallowStream uses about 2.6 s of compute, HERMES 6.2 s, OASIS 50.4 s. Long-cluster holds peak GPU memory near 18 GiB from 64 to 1,024 frames; the uncompressed archive grows to 21.76 GiB. The gate costs 92.7 ms and evidence selection 106.0 ms, a small slice of the 1.759 s full query.
In retrieval ablations, token voting beats pooled shallow Q-K and a standalone SigLIP encoder. Max-min diversity then stops the budget from collapsing onto near-duplicate shots.
The steady-state cost of a streaming system is the price of each arriving frame. Cutting prefill from full depth to the first 4–5 layers is a bet that most frames will never be named by a future question. For wearables, monitoring, and other always-watch, sometimes-ask settings, that asymmetry is a systems problem, not an offline long-video problem with a causal wrapper.
The method is a training-free inference rewrite and plugs into Qwen3-VL-8B and LLaVA-OV-7B. Anyone already running ReKV or HERMES-style full-depth KV can treat this as a cheaper index: shallow layers retrieve, deep layers run only on the hits.
The accuracy edge over OASIS and SimpleStream is small. Compute and latency are where the gap opens. This is not a new backbone.
The paper is marked Work in Progress and has no standalone limitations section.
On the synthetic calibration set, the LLaVA-OV gate hits 97.78% precision and only 44% recall. High precision, low recall means many retrospective questions get routed as "recent is enough." Its OVO backward average is 49.0, a touch below SimpleStream at 49.6; real-time perception is what lifts the overall score. Qwen3-VL gate recall is 97%, so this failure mode sits mainly on the weaker backbone.
The 52.1× figure is an "up to" against the heaviest path, which is OASIS-style answer-level routing. Versus HERMES at an 80-second interval the gap is 6.2 s vs 2.6 s, about 2.4×, not 50×. Memory numbers are peak GPU only; the shallow archive lives in host RAM, and there is no combined CPU–GPU curve. On StreamingBench, LLaVA-OV counting (CT) is 35.2 against HERMES at 45.6.
Generation is capped at 16 new tokens, so the benches are short-answer. The gate threshold comes from 200 synthetic questions. Keeping eval queries out of calibration is clean; how well that synthetic mix matches real users is untested. The "shallow is enough" claim is measured on two 7B/8B backbones under bounded local-attention stream prefill. Deeper models, or dense full-history attention, may want a later cut. Long-cluster collapses a cluster to one representative, so misaligned fine-grained moments can disappear.